In some ways, a screen adaptation make less sense with every passing year.
I normally agree, but with AI agents I find myself thinking about Neuromancer frequently. Somedays I feel like a Netrunner with what my AI and I can do. I am also now getting slightly paranoid of what other people and their AIs can do. I bought a 'deck' (GPU) for the first time in a decade and my office glows with color and once again has the reassuring sound of whirring fans. I've attempted to secure my network like I never have before and have my AI watching over it. It can take some automated actions but not enough for it to automatically defend...yet. Maybe I feel more like a shitty Netrunner at this point :)
Even posts about hacking into your own air purifier[1] make me feel like we have the power again. Some of that naive? 90's era optimism. Regardless, I hope the show is good!
That is kind of the point of this repo. If your API implementation is too expensive for lambda move the same container to run on EKS/ECS fargate or ec2. The opposite holds as well - if you can save money by moving it from fargate to lambda then you can use the same container.
Ion has a binary format but is not specifically about real-time streaming. It is a JSON replacement.
Ion originated 10+ years ago from the Amazon catalog team - the team that kept data about the hundreds of millions of items available on Amazon. Nearly every team in the company called the catalog to get information about items all the time - scanning the entire catalog, parts of the catalog, millions of individual item lookups every second, etc.
They did the math and some very large percentage of network traffic in Amazon Retails data centers was catalog data. If that data, currently in XML or JSON format, was sent in a more compact format it would save some ridiculous millions of dollars every year. So Ion was born and eventually open sourced.
The intent of the stored schema isn't really for self-description. A typical use case for Avro is data storage over long periods of time.
It is expected that the schema will evolve at some point during this time. Therefore you still need to specify a target schema to read the data into which is allowed to be different than the stored schema. Avro then maps the stored data into the target schema by using the stored schema. Most avro libraries expect you to get the target schema from a separate source before reading data.
That's not true. Avro at least existed at the time. However they wanted something self-describing to replace JSON/XML usage. Avro is better suited as a data storage format rather than a transit oriented format. Of course, both Ion and Avro can be used for either, but Avro will give you better compression on disk, but Ion is less cumbersome since it doesn't require a schema
Finally! I've had to live the JSON nightmare since I left Amazon.
Some of the benefits over JSON:
* Real date type
* Real binary type - no need to base64 encode
* Real decimal type - invaluable when working with currency
* Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo').
* Text and binary format
* Symbol tables - this is like automated jsonpack.
* It's self-describing - meaning, unlike Avro, you don't need the schema ahead of time to read or write the data.
Sounds a lot like Apple's property list format, which shares almost everything you listed in common, except for annotations and symbol tables.
Its binary format was introduced in 2002!
Edit: Property lists only support integers up to 128 bits in size and double-precision floating point numbers. On top of those, Ion also supports infinite precision decimals.
Plists are nifty, but the text format's XML-based, which makes it too complex and too verbose to be a general-purpose alternative to something like JSON.
(plutil "supports" a json format, but it's not capable of expressing the complete feature set of the XML or binary formats.)
Like Property Lists the binary format is TLV encoded as well. Ion has a more compact binary representation for the same data and additional types and metadata. Also, IIRC, Plist types are limited to 32-bit lengths for all data types. The binary Ion representation has no such restriction (though in practice sizes are often limited by the language implementation).
True, but my point was that there's enough talent at Amazon, working on SDKs, and others, and there are precedents where even more complex projects such as JMESPath have wide support [0].
I'm sure there are ion bindings for every language in common use at Amazon. But a huge percentage of Amazon code is Java, so presumably this one was the best maintained and documented.
I doubt it, when I was there Ion was only used by only a handful of Java teams doing backend work. It was also horribly documented and supported at the time (3.5 years ago).
I am still in Amazon and Ion is definitely the most widely used library around. It has among the best documented code and some of the extensions that have been built on top of Ion are simply amazing.
Precision is the number of significant digits. Oracle guarantees the
portability of numbers with precision ranging from 1 to 38.
Scale is the number of digits to the right (positive) or left (negative)
of the decimal point. The scale can range from -84 to 127.
Worth noting this isn't specifically an Oracle thing, most financial systems need to be sure that it can store currency numbers accurately and this convention is widely used to ensure this.
Typically when dealing with currencies scale is only used to represent the units less than whole unit of the currency, i.e. cents and pence. But there isn't anything that restricts it from being used to accommodate larger numbers with the use of negative scales.
<CcyNtry>
<CtryNm>UNITED STATES OF AMERICA (THE)</CtryNm>
<CcyNm>US Dollar</CcyNm>
<Ccy>USD</Ccy>
<CcyNbr>840</CcyNbr>
<CcyMnrUnts>2</CcyMnrUnts>
</CcyNtry>
The CcyMnrUnts property denotes 2 decimal places for the sub-unit of the dollar.
So for the above example of 99999.999 you would store an amount of 99999999 and a scale of 3.
So that it can be an integer with an arbitrary length, or a float/double without precision problems. You can also let our own integer classes do the parsing (which might even be able tonhandle complex types).
After all, everything in JSON is a string since it doesn't have a binary format and it shouldn't cause a huge overhead to do the parsing yourself (that might depend on the library, though).
I find that many financial technology companies opt to store currency as strings. The small overhead is typically well worth freedom from floating-point errors.
Any text format is a technically a string. Just because some numeric token has no double quotes around it doesn't mean it isn't a string (in that representation).
It's just that we can have some lexical rules that if that piece of text has the right kind of squiggly tail or whatever, it is always treated as a decimal float instead of every programmer working with it individually having to deal with an ad hoc string to decimal type conversion.
Until you need to deal with decimals instead of floats, then you are going to hate yourself because you have to pull in some third party library because the language treats every single number as a float (and floating point errors are a lot more common than most people think even when they are adding together simple numbers).
Sure, but most other languages have built-in support for decimal types. Java has BigDecimal, as does Ruby, Python has the decimal module, C# has System.Decimal, the list goes on.
Javascript doesn't even have proper integers to guarantee the functioning of this correctly, it's a really sad state.
I just did some ownership percentage stuff where it's not uncommon to go 16 decimal places out...working with JavaScript on this was a pain. Never thought I'd care about that .00000000000001 difference hah...
Ion has functions to turn Ion into JSON which will, of course, lose information. Annotations are dropped, decimals turn into a JSON number type which may lose precision, etc.
To be honest, we weren't really sure what to pick. We just wanted to have something in place. It's not set in stone and we'll likely change things as we see different use cases.
I normally agree, but with AI agents I find myself thinking about Neuromancer frequently. Somedays I feel like a Netrunner with what my AI and I can do. I am also now getting slightly paranoid of what other people and their AIs can do. I bought a 'deck' (GPU) for the first time in a decade and my office glows with color and once again has the reassuring sound of whirring fans. I've attempted to secure my network like I never have before and have my AI watching over it. It can take some automated actions but not enough for it to automatically defend...yet. Maybe I feel more like a shitty Netrunner at this point :)
Even posts about hacking into your own air purifier[1] make me feel like we have the power again. Some of that naive? 90's era optimism. Regardless, I hope the show is good!
[1] https://scottjg.com/posts/2026-11-11-reverse-engineering-my-...
reply