The best kittens, technology, and video games blog in the world.

Showing posts with label json. Show all posts
Showing posts with label json. Show all posts

Saturday, August 01, 2026

The Ultimate Fix for JSON format

image by bing image generator

I previously wrote how JSON is a fake format, which leaves important parts unspecified, so sending JSON from Python to Python, or from JavaScript to JavaScript generally works, but if you mix languages, results are largely unpredictable.

So, should every programming language have its own JSON-like, expressing data types it supports faithfully? That would also be a huge mess, so here's my proposal instead.

There is no code included, but code for it would be fairly straightforward if anyone wanted to go forward with it.

Baseline

All current JSON would still parse under this proposal, but unfortunately all existing ambiguities would still remain.

The proposal is primarily meant for machine to machine communication, but any such format always ends up being written and edited by humans as well. Therefore, before we go any further, we need to include JSON5's obvious fixes. Comments and trailing commas are a must, but there's no downside to just making JSON5 our baseline, and supporting plus and minus infinities, NaNs, and single quoted strings. Technically, this is orthogonal to the rest of the proposal, but if we have an opportunity to fix known issues with JSON, we might just as well do it now.

Extensions mechanism

And now here's the key part - just add one extra construct TypeName(arguments). We don't need hundreds of different syntaxes, this covers everything we might possibly want. Different languages will use a different subset of them, but the important thing is that most of these can be widely shared.

Extensions represent data types semantically, and their names won't necessarily match whatever a given language calls that data type (Time vs Timestamp vs DateTime etc. - just pick whatever's clearest).

Universal extensions

These two types are representable in pretty much every even halfway sane language that exists, and they're badly missing:

Time("iso string") - this is supported by pretty much every language. Everything not in standard timestamp format is an error.

Integer(n) - arbitrary precision integer, native Integer type in Ruby/Python, BigInt in JavaScript. This represents a mathematical object, so for this type Integer(0) and Integer(-0) are defined to be the same. Everything that's not an integer as argument is an error.

Other extensions

Depending on your language, you could have a lot of other useful types, like:

  • Date("YYYY-MM-DD")
  • Map([[key, val], ...]) - with arbitrary keys, not just strings
  • Set([elements])
  • Decimal(arbitrary precision number)
  • Double(exactly IEEE 754 floating point)
  • UUID("hex string")
  • Rational(num, den)
  • Bytes("base64 string")

There could be even ones that are limited to very few languages, like:

  • Symbol("string") - for Ruby symbols, Elixir Atoms etc.
  • Tuple([elements]) - for Python tuples
  • UInt128(number) - for languages that really need specific precision

Some group of common extension types would likely be agreed on, and these should be generally representable in most languages. We probably won't see round trip invariance - parsing Date(...) might convert it to Time(...), parsing Symbol(...) to a string, parsing Tuple to an array and so on.

Defining new extensions

If you're developing a new programming language with some extra datatypes, or you just have some user-defined types that you really want to represent, you could still use the same format with the same parser. The parsing library could offer hooks to pass whitelisted Type(...) constructors to your hook. These would generally not be interpretable if you send it to another environment, but such data would still parse, and a system could let it stay uninterpreted, or have some universal fallback like converting it to a string.

What should stay out of scope

To keep this format from getting overcomplicated, I'd recommend excluding some things:

  • any circular object graphs
  • any code-like objects including Regex()

And for obvious security reasons, the parsing library should not just try to parse unknown types by matching classes in the current environment by name and hoping for the best.

Validation

Any such format can have schemas defined, with type and other validations. The only nontrivial part is handling any types that are coerced. For example, if in your Python reader Symbol("foo") and "foo" are going to end up as "foo" anyway, do you want to specify that field as Symbol|string (pre-coercion), or as just string (post-coercion)? I'd lean toward the latter option.

Monday, July 27, 2026

JSON is a fake data interchange format

python picture by Grok

It must seem really strange to complain about JSON in 2026, but I never got over how fake it is.

Data in a JSON file is supposed to represent something regardless of programming language used, but does it really?

Zeroes

Let's start with this simple JSON - [0, -0, 0.0, -0.0]. What is it, and which values here are distinct, and which are the same.

In JavaScript this parses to:

  • float 0.0
  • float -0.0
  • float 0.0
  • float -0.0

In Python or Ruby this parses to:

  • int 0
  • int 0
  • float 0.0
  • float -0.0

Does JSON 0 mean the same thing as -0, or the same as 0.0? Well, nobody can agree on that, because JSON only fakes it.

Even worse, just passing this JSON through a pretty printer can change it arbitrarily.

json_pp (part of Perl, preinstalled with OSX and most other Unix systems) returns [0, 0, 0, 0]. Which matches neither JavaScript nor Ruby/Python.

json_xs (also part of the exact same Perl, preinstalled with OSX and most other Unix systems) returns [0, 0, 0, -0] instead. This isn't even idempotent, passing it through json_xs twice gets us to [0, 0, 0, 0]!

jq preserves all numbers at least. This online pretty printer turns it to [0, 0, 0, 0]. This one goes with [0, 0, 0.0, -0.0].

Pretty printers are quite trivial, but data sent from one program to another can easily go through some database or proxy or something else that would mangle it.

And how other languages do it? Most follow Ruby/Python, but some go for JavaScript, and good luck guessing which way it goes.

Big integers

OK, so you don't care for zeroes. Let's look at something that matters a lot more - big integers.

3**39 = 4052555153018976267 is not even anything special, just a 64-bit integer. Can JSON represent it? Well, it depends.

Ruby, Python, json_pp, and json_xs all let it pass through correctly.

Some random pretty printers? This one mangles it. As does this one. As does JavaScript, despite big integers existing in JavaScript.

But what if we try 3**80 = 147808829414345923316083210206383297601 which is a 128-bit integer? The two online ones above mangle it to a double again, but interestingly json_pp and json_xs turn it into a string now! It's "147808829414345923316083210206383297601", not even a number, but at least it's not rounded.

Big decimals

Up to this point things sort of work if we avoid JavaScript, Go, and other broken languages. But now it gets worse for all languages.

In principle big decimals are 100% representable in JSON. Does any language even try?

Ruby will represent them as strings, weirdly "0.33333333333333333333333333333333e0". Python will just refuse with Object of type Decimal is not JSON serializable.

If you actually pass 0.33333333333333333333333333333333 through any pretty printer other than jq, it will round it to double precision, or something close to it.

Every language has its own JSON flavor

And so, JSON is a lie. JavaScript JSON and Python JSON are not the same at all! Ruby actually seems to match Python perfectly, in a rare interop success story.

Other languages might match one of these two, but usually they do something slightly different, like some do int32+float64, some do int64+float64, Rust somehow does "I'm not even going to try, here's a bunch of accessors, figure it out by yourself" (but even that up to 128-bit only, no bigints).

And what's worse - your data might go through proxies, databases, and other steps that can mangle it.

This isn't anything exotic - integers bigger than 32 bits are a very common data type.

Is it just numbers?

Numbers are the biggest area of disagreement, but there's another with objects.

Depending on implementation, order of fields might or might not be preserved. It's not so much an interoperability problem as it is a headache as serializing an identical object might result in multiple different JSON representations.

Surprisingly, the standard doesn't actually ban duplicate keys with the same name! It only says that "The names within an object SHOULD be unique", and advises against using duplicate names. It's been a very long time since I've last seen duplicate keys in the wild.

And technically there's one more area where the standard admits implementation differences with handling Unicode, but I haven't seen that in the wild either.

The Mystery

Here's a mystery I could never quite figure out. If JSON is really programming language specific anyway, why doesn't each programming language define their own JSON-like format, to express what it can represent? Beyond solving numbers, it would at least have date and time objects, in JSON typically escaped as strings. It could have proper big decimals, tuple/array indexed objects, and whatever extra stuff like symbols a given language uses.

And since we're at it, they could also add comments to the format.

As far as I can tell, no language ever seriously tried to do this, they all just use their own flavor of JSON and pretend like it's standard.