Yup, I think the same applies to traffic of arms regulations.. Private firms sell to countries they are 'allowed' to, passing it between organisations in a chain until they get to the original customer they couldn't sell to directly.. Cuts are taken along the way, and people making money look the other way..
If call centres are anything to go by, it will be penalized to work on a single thing at once. The longer your workplace can resist measurements related to agents, the more sane the workplace will remain IMO, otherwise, it will turn into what happened to call centres around the 2000s. Measure everything down to the second, and introduce expectations that you will need to spend time justifying if you miss them.
IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve.
Strata, Radiance, Maya, etc have really changed that IMO. I use an old X99 setup with 2x A4000s and get a solid ~70 tk/s decode, ~2k prefill, and with 128GB 2133 DDR4, I have memory for quite a few sessions worth of cache so multiple agents can run with decent switching. Probably other performance nobs to setup, but I've found this is great with Qwen 3.8 Flash Next IQ3_S .
Yes, hardware has gotten insanely expensive, but software is moving so quickly and a lot of effort is pointed at eking out every bit of performance, and we are seeing a lot of success. Really looking forward to Qwen 4+. Yes Opus 5.5 is better than what I can use at home, but the way I use these models, I don't feel I need that. I prefer to be in the drivers seat, and these delegate the simple tedious parts to models which they are getting really good at.
I have real world data here.
On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.
This is totally suitable (speed wise) for coding.
This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).
No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token.
Totally agree, but also deeply flawed human being that he played so well. My favorite writing of Picard and Stewart's performance was those times that showed the toll of always being that man. But despite those personal costs, to always try to take the compassionate view, and fiercely reject those that try to oppress others.
Software is improving quickly so take these numbers as a snapshot performance at best. Strata has really shown how much performance is left on the table. I have a dual A4000 16gb (Ampere) on an old X99 dual Xeon E5-2680 v4 with 128gb of memory and Qwen 3.8 Flash Next at the IQ3-S size (~96Gb) runs at 65-75 t/s which is extremely useable.
reply