Local models already crossed the threshold where they're quite good and useful for many things, even with cheap setups. I walked away really impressed with (non-coding) use of Byteshape's Qwen3.5-9B quant on my 8GB RTX 3070 for instance. The model was fast, helpful, and 100% private.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
I don't think "useful for many things" qualifies as "winning". It's about two orders of magnitude shy of winning.
> and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research
Maybe. But competitive markets pay exponentially more for incrementally superior services. Professional athletes are a great example of this. If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
If you're bitter lesson pilled then the alpha is entirely in having a better model, more data, and more compute. Your vision is that human ingenuity will provide more alpha. Oof. I wish. But my tastebuds are very very bitter.
> If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
i highly doubt this premise. the success of a product has very little to do with the amount of the intelligence used in producing it. you could throw millions of dollars using AI to develop a movie and it would still flop in comparison to an indie-made film.
I think the premise is confusing what the amount even is in this context.
I want to use "more intelligence" but not in terms of single higher IQ.
I want to use more intelligence in terms of a sum quantity of knowledge work being performed. A team of 10 instead of a team of 5.
This is when a local model will shine. A fixed cost and I don't have to worry about how much I am using it. I already can't tell any difference at all between Opus and Fable. It won't be that long that I won't be able to tell the difference between a local model and Opus.
There will be an infinite demand for "more" intelligence but I think many people are making a category error on the dimension in question.
Also - the “open” here doesn’t have the advantage of open source, where it can be improved by the community. At best, the community can tweak it a little. For actual improvements you need to wait for the next model which is trained by companies for billion of dollars, and could stop releasing them at any moment.
It can, through better quantizations and reinforcement learning.
Just this week, qwen 3.8-27b got two new interesting quantizations, one of which came from community. Llama.cpp also improves continuously, and novel alternatives like Strata pop up.
The point is that these are overall small improvements that the community can’t really build upon. For an illustrative example, combine all the community improvement to the best open weight model released a year ago. It won’t do much to make it competitive or even interesting.
3.8 wasn’t released a year ago. Of course base models change. But the quantization techniques carry over to new models. For instance, one of the improvements last week was someone porting the techniques used for macaq to qwen3.8-27b. It’s pretty satisfying to get a working implementation of a new technique barely one week after the original paper hit arxiv.
Also improvements to llama.cpp stay, and many small improvements stack up nicely.
My Android TV has a non-default launcher with no ads, and I can sideload apps to get things like Youtube with sponsorblock baked in. Going to Apple's locked down system seems like a huge downgrade to me.
Depends on one’s needs. The overwhelming majority of Android TV boxes and integrated Android TVs use extremely anemic hardware that will choke on any content that falls beyond hardware acceleration support for their SoCs. So fine for YouTube and Netflix probably, but not so great for a personal Plex server full of HDR content.
Google can shuttle network requests to any of their other apps (or play services, if that's running) via IPC, even with the network permission disabled in Gboard.
So if you're using Graphene for privacy from Google specifically, it's not safe to install things like Gboard even with the network permission disabled. Even if they don't use IPC in this way today, there's no telling what the Google-of-tomorrow will do.
there's no telling what the Google-of-tomorrow will do.
This is true and even vastly understated. Google is like the Walter White of companies. Starts out innocent, turned into a monster, everyone around him astonished at the dark, depraved change.
So if one thinks "nah, Google wouldn't do that", I have a bridge to sell you...
Like I said, it's impossible to "defeat" it outright because the concept doesn't really lend itself to this sort of structure. If I buy a plaque with my name on it , therefore owning it, the only thing that could really keep me from selling it would be some centralized authority. Even then, black markets have existed for drugs for centuries.
Yes, all this talk about the language used still leads us to the same frustrating outcome: customer pays for something, and at some unspecified point in the future that same thing is taken away from them. I don't care if 6 years earlier I clicked a button that said Pay vs Buy, I care that my purchase is being removed due to some backroom licensing issue.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
reply