Shotout to PowerSync for enabling companies to go in the opposite direction and build offline-first apps. My company is a (production) customer, and we recommend it.
> Companies want everything to be push instead of pull.
This reminds me of Instagram's search page, which is full of animated videos jumping at you to dissuade you from completing the search you wanted and just click on what they want you to see.
Social media sites are full of this. They want to decide what you see or don't see instead of just showing you what your friends post in chronological order. Streaming services are another good example. They don't want to give you an A-Z listing of what's on offer. They want to show you a small selection of titles, repeated over and over across a bunch of vague/meaningless categories.
The last big redesign netflix made was pretty much universally hated, but they were mostly upfront about the fact that it was intended to take choice away from users and highlight the shows they wanted to promote. For a company that got where they are because people wanted "on demand" over cable TV it seems strange to hear them now saying "Choosing for yourself is so hard! Just let us tell you what to watch!" but that's the attitude every company seems to be taking.
Or every social media (especially Instagram)'s home page which shows the most engagement-baiting videos of this week and nothing from the people you follow
Disappointed, since I was expecting they would rewrite the implementation from C++ to Zig. I bet that would increase the number of positive pull requests they get, since most developers prefer to stay away from C++ nowadays.
I see many comments comparing PgBouncer with application connection poolers without addressing the conceptual difference between them. Here it is:
1. Most application connection poolers follow a first-in-first-out (FIFO) algorithm, which is simple enough to implement and is enough to make sure the application always has a connection available to connect to the database. It optimizes low latency, and works great from the point of view from the application. The problem is that it has few mechanisms to remove redundant connections, since the application is constantly keeping them all "warm".
2. PgBouncer and very few external poolers follow the inverse idea – last-in-first-out (LIFO), and they optimize for reducing the number of connections that reach Postgres, thus improving its throughput. The idea might seem crazy at first – the last connection used is the first one to be picked up again – but this algorithm automatically removes excess connections, which will get cold and get closed.
When starting a new application, option (1) is enough, but as it scales up enough, at some time it is recommended to use (2), since having hundreds of open connections to Postgres is bad for performance if you can use PgBouncer or similar to cut it by 90%. Postgres' process-per-connection design works much better when there are fewer connections reaching it.
Not necessarily. You design your data model to prioritize (1) correctness and (2) performance. It doesn't have to resemble the UI at all, as long as the UI can fit on top of it with some abstractions.
Some examples that come to mind:
- video games with their entity-component systems;
- high-performance text editors like VS Code. [1]
I wasn't advocating a single-layer data model or speaking against common data structures, just saying the structure of the data should follow directly from user goals.
Correctness and performance are normally the top user goals.
I wonder how we could handle that in a simpler way with durable workflows (e.g. Temporal, Restante, DBOS) – which are similar to Erlang processes but with persistent disk storage. This could avoid the need to maintain the 1000 row inventory.
Perhaps each shopping cart would have its own workflow, and the inventory item would have one as well. Then, whenever a customer put an item in their cart, their cart workflow would send a signal to the inventory item workflow and wait for the response. The inventory item workflow would maintain a ledger controlling to which cart each unit goes, and it could batch the writes to this table. This way, even if 100k customers try to purchase the same item in the same second, it should handle the load.
After the batch is written to the ledger, the inventory item workflow would reply signals to each cart workflow confirming that the reservation was completed. The end-to-end latency from the consumer point of view would be a fraction of a second, without needing the 1000-row hot-inventory heuristic.
Durable workflows are different, not necessarily simpler, imo. Unless the team is already familiar with them, I wouldn’t introduce one just for this. You also have to account for the infrastructure needed to run and manage the durable workflow itself, which adds complexity.
Instead of spending all of this effort trying to optimize an OLAP data lake for an OLTP use case, why not ust use a regular OLTP database like Postgres or MySQL?
Even if the data size is "infinite" you can put it on PlanetScale or something like that. If the tables are well optimized with covering indexes, you can pull out thousands of rows in 100ms for point queries, which is plenty for "point" queries.
1PB of EBS volumes on sc1-class HDD is roughly $15k/month.
Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.
These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.
Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise.
That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.
Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index).
With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.
