Interesting article... my 2c is that you might want try a library like async.js, that handles asynchronous code with explicity flow-control based structure. Async has throttling ( https://github.com/caolan/async#queue ) and all kinds of things built into it that can address some of those things.. for other things where you are worried about buffers, you could use things like RabbitMQ or AWS-SQS to mitigate situations where you have mismatch in push/pull.
async.js still suffers the lack of real flow control that the article is talking about. In fact that queue you linked to is a perfect example of a naive implementation with no back pressure at all.
The question is: how do you guarantee that your queue never grows beyond a certain size? The library doesn't even attempt to help with that. And anything that you implement yourself would still suffer the race condition mentioned in the article.
Agrred. I want to add that this is why systems engineering is its own thing. Application developers seldom understand what's going on at the copying bytes around/syscall level.
People sometimes mock the idea of "rockstar" developers, but this is one of the things that separates the merely good from the great.
A developer who can jump up or down four levels in the technology stack with ease is an entirely different species, when compared with someone who lives at one level.
