If your goal is to prevent DoS attempts from degrading the service of other tenants in a multitenant environment, fair queuing is the optimal approach. Give each client their own queue to which incoming traffic is enqueued, and have a background routine that repeatedly iterates over each queue, dequeuing a single request and servicing it. Any client that spams requests will only congest their own queue and not those of other clients.
At some point you have to stop clients from enqueuing further requests, you can't grow the queue indefinitely. At this point, isn't it equivalent to rate limiting each client and a shared queue?
Not quite, because it isn’t a fixed rate limit. Suppose you can handle 120 requests/second across all customers. If you have 3 clients with active requests, they each are being served 40 requests/second, even if one of them has filled up their maximum pending requests. If you have 6 clients, each are being served 20 requests/second.
If you are applying a fixed rate limit to each client, you’d need to adjust the rate limit dynamically based on the current number of clients in order to reproduce the same behavior.
This eliminates the benefits of multitenancy since you don't get to downsize the server - you still provisioning for maximum rates on all clients simultaneously
I'm not sure I follow the argument. The scheduling seems like it would be de-coupled from the provisioning. With the static rate limit, if the only additional requests are from the high-rate client, then those requests are ignored and the server is idle. With the fair scheduling, the server is only idle if all requests from all clients are filled.
Which is beneficial depends on how the payment scales. If clients pay for access up to some rate limit, then the rate limit is there to enforce payment, and serving additional requests above that rate limit is an additional cost without a benefit. If clients pay per request, then any rate limits are there to ensure quality of service, and serving additional requests above the rate limit is additional revenue.
Huge disclaimer here: I'm one of the founders. You should most definitely be able to set concurrency levels, throttling, rate limiting, etc. per your own tenant, in code, without having to mess around with creating independent queues or streams for users, and without managing state. We let you do that.
We usually implement queueing on the route to those specific things that are vulnarable. If you can not discern what traffic does what, you just need move the rate limiter close to the application or the problem hot spot. It's perfectly valid to give a HTTPS response 429 from a backend and let your frontend handle that in some graceful way. The same is valid as an exception in code, the nearer the problem spot you get the harder it is to get right.
In the abstract sense instead of pulling from queues round-robin you can assign "tokens" to each queue round robin. When the number of tokens a queue has is equal to the cost of the request reset the tokens and pull that request.
This can also be used to handle priority. Maybe paying customers or customers on the enterprise plan get 2 tokens per round or their requests only have half of the cost.
