It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat.
Not sure if the benchmarks have been run recently or not but I’m definitely curious how things stack up to this post from 6 years ago :)
I actually ran them in July 2019 against QueueClassic using the Que benchmark;
queue_classic jobs per second: avg = 1879.1, max = 2072.1, min = 1779.4, stddev = 128.3
que jobs per second: avg = 1500.5, max = 1550.8, min = 1405.2, stddev = 58.1
I wrote django-postgres-queue for this purpose. It uses postgres transactions to keep queue and application state in sync. It also uses SKIP LOCKED to avoid some of the typical issues with using a database as a queue.
Recently was wishing this myself. I’ve been using rq, which although I hear is much faster than an rdbms is also another dependency and service to worry about failing. Trying to keep things simple is nice, I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains
> I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains
If it is possible to set up a smaller test environment with one or a few instances of your DB and Redis, and you have time to do it, you could do some testing where you purposely shut down each of them at various points in time and inspect what happens to your application state compared to how it behaves when all is well.
Perhaps you could even outfit the testing version of your application with two proxies, one that will proxy the DB connection and one that will proxy the Redis connection, and have these proxies randomly decide within some threshold whether to pass the data along to their upstreams, or to simulate a broken connection. Then test the application with some different threshold values, for example, 1% probability of failure, 5% probability of failure, 50% probability of failure and 100% probability of failure. Repeat the tests some number of times for each threshold value and observe the behavior of the application each time. Also, make sure that you log the decisions made by the proxies each run, so that you are able to look at these afterwards.
I mean, I love testing, but this is a great example of why you should just use less things if you can. Combinations of failure modes aren't fun to get right.
It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat.
Not sure if the benchmarks have been run recently or not but I’m definitely curious how things stack up to this post from 6 years ago :)