I wrote this post, summarizing years of working with scientific data archives across multiple fields of science.
Everyone wants to make their scientific data archive work with Agents and AI, but this requires a complete rethink of the storage layer. Luckily, the recommendations end up being very similar to what data providers should have been doing anyway: forget about files!
This post takes you from "what is a chunk" all the way to understanding why there is really only one sensible way to manage large scientific datasets in the age of AI.
It's also a choose-your-own-adventure post - I didn't want to have to write the same thing 5 times for 5 different fields of science, so I just made one post in which you can toggle examples between different fields!
Whilst I weakly agree with this characterisation, it's not at all fair or accurate to put epidemiology and climatology in the same category as your other examples. Yes they have some weaknesses in replication practices (especially for purely computational work), yes the feedback loop for self-correction can be slow, but it would be a mistake to conclude or imply that the major results of those fields are therefore unreliable.
That's because those fields do still make objective empirical predictions about the world, which might take years to be testable but are still ultimately testable. For example, it's been convincingly shown that climate models from decades ago predicted our current climate pretty well. [0]
These fields are more comparable to subfields of physics which only get to do big experiments a handful of times, e.g. because they can only observe so many planets/supernovae/universes, or because they have to spend decades building a new billion-dollar machine to test new hypotheses.
We should also not confuse these arguments with uselessness - soft social science work can still be societally valuable without being easily empirically testable.
Disagree, those two fields are extremely unreliable. It takes a lot of things happening simultaneously to be scientific. Yes, those two do make testable predictions, sort of. But that isn't enough.
Modern (computational) epidemiology is rife with unscientific practices. They ignore data that shows a model was invalidated so the fact they make testable predictions isn't really useful. They also engage in a lot of circular reasoning, buggy coding and logical fallacies. During COVID I wrote a whole report on this topic for some politicians [1]. But the biggest issue is "A problem in theory" again - epidemiologists conflate fitting a curve in R with developing a hypothesis, so the field is overrun with overfit models that aren't based on any refinable theory of disease, just misuses of statistics. Even if you prove a paper's predictions were wrong it changes nothing because nothing built on it anyway.
The problem in climatology is that when the models don't fit the data they just change the data and claim victory, e.g.
> The problem in climatology is that when the models don't fit the data they just change the data and claim victory
That is... not remotely true, nor supported by either of the links you shared. (Nor by any cursory look at the state of the world today, with well-anticipated Himalayan glacial floods as front-page news.) You therefore appear to be leaping over the line from healthy skepticism to irrational conspiratorial thinking, so I'm going to stop engaging, sorry.
The two links I give show the problem in action. In 2013 the IPCC was concerned because temperatures had gone sideways for a decade. Der Spiegel's first sentence is "Data shows global temperatures aren't rising the way climate scientists have predicted" and the next says they were trying to "hush it up". In 2015 climatologists announced a new temperature record that replaced the entire 21st century with different data, as reported by Nature.
That's an unambiguous sequence of events. You can't make statements about replicability in an environment where they routinely decide their already published data is wrong post-hoc, without retracting the papers built on that data.
Anyone interested in accessing GOES data at scale will find this interesting - I created a Zarr index over the 7 billion chunks of data in the GOES-16 archive.
Yes! But if you want to share it with anyone else that would be great, since we're advocating for fairly radical changes within a big bureaucracy here, as I'm sure you will appreciate :)
The last 2 years of my professional life have in some sense all been working towards this blog post.
It shows how you can make an enormous archive of public scientific data[1] much easier to use for everyone, without having to copy all the data, which would be prohibitively expensive at this scale.
I also made some sweet gifs and images of the Earth from space from the raw data so check those out!
[1] In this case satellite data, but this could work for data from many other fields - my background is original nuclear fusion plasma physics then oceanography.
Awesome to see this project here - it was partly inspired by my blog post (original is linked from the OP, but there's a slightly newer version on my personal site here[0]).
What you describe is very similar to how Icechunk[1] works. It works beautifully for transactional writes to "repos" containing PBs of scientific array data in object storage.
The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0].
Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching the metadata up front so you know where the chunks you want are [1].
The data model of Zarr [2] generalizes this pattern pretty well, so that when backed by Icechunk [3], you can store a "datacube" of "virtual chunk references" that point at chunks anywhere inside the original files on S3.
This allows you to stream data out as fast as the S3 network connection allows [4], and then you're free to pull that directly, or build tile servers on top of it [5].
In the Pangeo project and at Earthmover we do all this for Weather and Climate science data. But the underlying OSS stack is domain-agnostic, so works for all sorts of multidimensional array data, and VirtualiZarr has a plugin system for parsing different scientific file formats.
I would love to see if someone could create a virtual Zarr store pointing at this WSI data!
IMO Zarr is that newer format. It abstracts over the features of all these other formats so neatly that it can literally subsume them.
I feel that we no longer really need TIFF etc. - for scientific use cases in the cloud Zarr is all that's needed going forwards. The other file formats become just archival blobs that either are converted to Zarr or pointed at by virtual Zarr stores.
Sounds like an approach that would also work for ML model weights files — just another kind of multidimensional array with metadata.
I wonder what exactly the big multi-model AI companies are doing to optimize model cold-start latency, and how much it just looks like Zarr on top of on-prem object storage.
People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is though.
It's definitely one of many fields that see convergent evolution towards something that just looks like Zarr. In fact you can use VirtualiZarr to parse HuggingFace's "SafeTensors" format [0].
> Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks
Yeah, a recurring thought is that these should condense into Apache Arrow queried by DuckDB but there must be some reason for this not to have already happened.
God this article is 10000% better than the posted one. This is great:
> Names should not describe what you currently think the thing you’re naming is for. Imagine naming your newborn child "Doctor", or "SupportsMeInMyOldAge". Poor kid.
The pitch for this sounds very similar to the pitch for Vortex (i.e. obviating the need to create a new format every time a shift occurs in data processing and computing by providing a data organization structure and a general-purpose API to allow developers to add new encoding schemes easily).
But I'm not totally clear what the relationship between F3 and Vortex is. It says their prototype uses the encoding implementation in Vortex, but does not use the Vortex type system?
The backstory is complicated. The plan was to establish a consortium between CMU, Tsinghua, Meta, CWI, VoltronData, Nvidia, and SpiralDB to unify behind a single file format. But that fell through after CMU's lawyers freaked out over Meta's NDA stuff to get access to a preview of Velox Nimble. IANAL, but Meta's NDA seemed reasonable to me. So the plan fell through after about a year, and then everyone released their own format:
On the research side, we (CMU + Tsinghua) weren't interested in developing new encoders and instead wanted to focus on the WASM embedding part. The original idea came as a suggestion from Hannes@DuckDB to Wes McKinney (a co-author with us). We just used Vortex's implementations since they were in Rust and with some tweaks we could get most of them to compile to WASM. Vortex is orthogonal to the F3 project and has the engineering energy necessary to support it. F3 is an academic prototype right now.
I note that the Germans also released their own fileformat this year that also uses WASM. But they WASM-ify the entire file and not individual column groups:
Andrew, it’s always great to read the background from the author on how (and even why!) this all played out. This comment is incredibly helpful for understanding the context of why all these multiple formats were born.
If I could ask you to speculate for a second, how do you think we will go from here to a clear successor to Parquet?
Will one of the new formats absorb the others' features? Will there be a format war a la iceberg vs delta lake vs hudi? Will there be a new consortium now that everyone's formats are out in the wild?
... Are you saying that there's 5 competing "universal" file format projects? Each with different non-compatible approaches? Is this a laughing/crying thing, or a "lots of interesting paths to explore" thing?
Also, back on topic - is your file format encryptable via that WASM embedding?
I would love to bring these benefits to the multidimensional array world, via integration with the Zarr/Icechunk formats somehow (which I work on). But this fragmentation of formats makes it very hard to know where to start.
Everyone wants to make their scientific data archive work with Agents and AI, but this requires a complete rethink of the storage layer. Luckily, the recommendations end up being very similar to what data providers should have been doing anyway: forget about files!
This post takes you from "what is a chunk" all the way to understanding why there is really only one sensible way to manage large scientific datasets in the age of AI.
It's also a choose-your-own-adventure post - I didn't want to have to write the same thing 5 times for 5 different fields of science, so I just made one post in which you can toggle examples between different fields!
reply