The Ultimate Fix for JSON format

I previously wrote how JSON is a fake format, which leaves important parts unspecified, so sending JSON from Python to Python, or from JavaScript to JavaScript generally works, but if you mix languages, results are largely unpredictable.

So, should every programming language have its own JSON-like, expressing data types it supports faithfully? That would also be a huge mess, so here's my proposal instead.

There is no code included, but code for it would be fairly straightforward if anyone wanted to go forward with it.

Baseline

All current JSON would still parse under this proposal, but unfortunately all existing ambiguities would still remain.

The proposal is primarily meant for machine to machine communication, but any such format always ends up being written and edited by humans as well. Therefore, before we go any further, we need to include JSON5's obvious fixes. Comments and trailing commas are a must, but there's no downside to just making JSON5 our baseline, and supporting plus and minus infinities, NaNs, and single quoted strings. Technically, this is orthogonal to the rest of the proposal, but if we have an opportunity to fix known issues with JSON, we might just as well do it now.

Extensions mechanism

And now here's the key part - just add one extra construct TypeName(arguments). We don't need hundreds of different syntaxes, this covers everything we might possibly want. Different languages will use a different subset of them, but the important thing is that most of these can be widely shared.

Extensions represent data types semantically, and their names won't necessarily match whatever a given language calls that data type (Time vs Timestamp vs DateTime etc. - just pick whatever's clearest).

Universal extensions

These two types are representable in pretty much every even halfway sane language that exists, and they're badly missing:

Time("iso string") - this is supported by pretty much every language. Everything not in standard timestamp format is an error.

Integer(n) - arbitrary precision integer, native Integer type in Ruby/Python, BigInt in JavaScript. This represents a mathematical object, so for this type Integer(0) and Integer(-0) are defined to be the same. Everything that's not an integer as argument is an error.

Other extensions

Depending on your language, you could have a lot of other useful types, like:

• Date("YYYY-MM-DD")
• Map([[key, val], ...]) - with arbitrary keys, not just strings
• Set([elements])
• Decimal(arbitrary precision number)
• Double(exactly IEEE 754 floating point)
• UUID("hex string")
• Rational(num, den)
• Bytes("base64 string")

There could be even ones that are limited to very few languages, like:

• Symbol("string") - for Ruby symbols, Elixir Atoms etc.
• Tuple([elements]) - for Python tuples
• UInt128(number) - for languages that really need specific precision

Some group of common extension types would likely be agreed on, and these should be generally representable in most languages. We probably won't see round trip invariance - parsing Date(...) might convert it to Time(...), parsing Symbol(...) to a string, parsing Tuple to an array and so on.

Defining new extensions

If you're developing a new programming language with some extra datatypes, or you just have some user-defined types that you really want to represent, you could still use the same format with the same parser. The parsing library could offer hooks to pass whitelisted Type(...) constructors to your hook. These would generally not be interpretable if you send it to another environment, but such data would still parse, and a system could let it stay uninterpreted, or have some universal fallback like converting it to a string.

What should stay out of scope

To keep this format from getting overcomplicated, I'd recommend excluding some things:

• any circular object graphs
• any code-like objects including Regex()

And for obvious security reasons, the parsing library should not just try to parse unknown types by matching classes in the current environment by name and hoping for the best.

Validation

Any such format can have schemas defined, with type and other validations. The only nontrivial part is handling any types that are coerced. For example, if in your Python reader Symbol("foo") and "foo" are going to end up as "foo" anyway, do you want to specify that field as Symbol|string (pre-coercion), or as just string (post-coercion)? I'd lean toward the latter option.