If you run it at home, it won't be realistically doing it 24/7. 8 hours per day is already 33% utilization. If the model is actually running half of that time, it's already 15%.
So, there will be always business for someone to host one in a data center, and sell the tokens at a markup, that will still be cheaper than buying something you will only use 15% of the time.
I don't think it's currently possible to calculate a reasonable valuation for an AI company using conventional techniques. It is too new, it is still rapidly evolving, nobody knows how it will reshape the industry. It's more of a poker game where the underwriters are betting on how much the investors are willing to throw into a perceived once-in-20-years oportunity.
There's a lot of use cases where you may not be personally interacting constantly but the LLM is running a lot of the time.
A consumer-hostable model like qwen3.8 flash next take longer than opus or better models to complete software builds, and I set mine off before bed, before work, and over lunch to take the next stage of some feature. It is running pretty consistently, just over more time than if it ran on the more powerful datacenter hardware that we use at work.
Let's say, you get 80 tokens/sec, so 80 * 3600 * 24 = ~7M tokens or $3.3/day at the current OpenRouter token pricing. That's ~2 years to break even at $2000.
That's also about 7kWh/day assuming 300W, so net $2/day or ~3 years.
Given how fast the hardware changes, it makes no sense at all.
Also, in my experience, Qwen 3.8 heavily relies on reasoning to produce meaningful results. So where Gemini-Flash would just give you a usable result in like 5 seconds, Qwen would stumble on a trivial technicality (like completely forget '+' and '-' in a patch), then reason for a minute or two on "hey, but what about that requirement, maybe it is related"? and eventually come to the right answer. But the speed and the token use is abysmal.
I mean, it's a cool model and it's a big step, and it's crucial to have open-weight AI, but currently it's not practical beyond proof-of-concept.
BTW, the current subscription situation is silly. The vendor harnesses force you into excessive token use, and then the subscription gives you a massive discount compared to OpenRouter pricing. So it looks like a good deal. But if you get more structured with what you want the AI in each specific task, $100 on OpenRouter can get you surprisingly high mileage.
Well, LLMs are priced per token, and most of the tokens are just echoing back the old code with minimal changes. So, a lot of the cost is actually paying for the LLM to echo back the same code.
Except, it's not that trivial to solve. I tried experimenting with asking the model to first give a list of symbols it will modify, and then just write the modified symbols. The results were OK, but less refined than when it echoes back the entire file.
The way I see it is that when you echo back the entire file, the process of thinking "should I do an edit here" is distributed over a longer span, so it has more room to make a good decision. Like instead of asking "which 2 of the 10 functions should you change" you're asking it "should you change method1? what about method2? what about method3?", etc., and that puts less pressure on the LLM.
Except, currently we are effectively paying for the LLM to make that decision for *every token*, which is terribly inefficient. So, there has to be some middle ground between expensively echoing back thousands of unchanged tokens and giving an error-ridden high-level summary. We just haven't found that middle ground yet.
I think the ideal way for these LLMs to work will be using AST-level changes instead of "let me edit this file".
grit.io was working on this years ago, not sure if they are still alive/around, but I liked their approach (just had a very buggy transformer/language).
They can, but this reduces the quality. The LLM has a harder time picking the first edit, and then all subsequent work is influenced by that one edit. Like first creating an unnecessary auxiliary type, and then being stuck modifying the rest of the code to work with it.
So, in practice, many tools still work on the file level.
For those not following: they are running inference on their own chips way faster than any competition, which I find really useful for refactoring some old code. It's like 3 seconds vs 40 seconds with Claude and the likes.
Their system had been hit or miss where where it would often get overloaded and kick down to slower speeds, but when it worked, it was nearly instant. Until they suddenly announced collaboration with OpenAI and took down non-OpenAI models, eventually pulling gpt-oss as well. I tried Gemini-flash in their absence, but it is nowhere as fast.
So, now they are back with the GLM and Qwen models. I wish the communication from them was better, and they would not leave users in the dark, but regardless, they have a wonderful product, and it's back online.
Bah. It's a really cool idea, but a rather crude way to measure the outputs.
If you just ask the model in plain text, the actual "decision" whether it detected anything or not is made by by the time it outputs the second word ("don't" vs. "notice"). The rest of the output builds up from that one token and is not that interesting.
A way cooler way to run such experiments is to measure the actual token probabilities at such decision points. OpenAI has the logprob API for that, don't know about Anthropic. If not, you can sort of proxy it by asking the model to rate on a scale from 0-9 (must be a single token!) how much it think it's being under influence. The score must be the first token in its output though!
Another interesting way to measure would be to ask it for a JSON like this:
"possible injected concept in 1 word" : <strength 0-9>, ...
Again, the rigid structure of the JSON will eliminate the interference from the language structure, and will give more consistent and measurable outputs.
It's also notable how over-amplifying the injected concept quickly overpowers the pathways trained to reproduce the natural language structure, so the model becomes totally incoherent.
I would love to fiddle with something like this in Ollama, but am not very familiar with its internals. Can anyone here give a brief pointer where I should be looking if I wanted to access the activation vector from a particular layer before it starts producing the tokens?
> I would love to fiddle with something like this in Ollama, but am not very familiar with its internals. Can anyone here give a brief pointer where I should be looking if I wanted to access the activation vector from a particular layer before it starts producing the tokens?
Look into how "abliteration" works, and look for github projects. They have code for finding the "direction" verctor and then modifying the model (I think you can do inference only or just merge the modifications back into the weights).
I am playing around with interactive workflow where the model suggests what can be wrong with a particular chunk of code, then the user selects one of the options, and the model immediately implements the fix.
Biggest problem? Total Wild West in terms of what the models try to suggest. Some models suggest short sentences, others spew out huge chunks at a time. GPT-OSS really likes using tables everywhere. Llama occasionally gets stuck in the loop of "memcpy() could be not what it seems and work differently than expected" followed by a handful of similar suggestions for other well-known library functions.
I mostly got it to work with some creative prompt engineering and cross-validation, but having a model fine-tuned for giving reasonable suggestions that are easy to understand from a quick glance, would be way better.
I haven't tried your exact task, of course, but I've found a lot of success in using JSON structured output (in strict mode), and decomposing the response into more fields than you would otherwise think useful. And making those fields highly specific.
For example: make the suggestion output an object with multiple fields, naming one of them `concise_suggestion`. And make sure to take advantage of the `description` field.
For people not already using structured output, both OpenAI and Anthropic consoles have a pretty good JSON schema generator (give prompt, get schema). I'd suggest using one of those as a starting point.
Adding entire files into the context window and letting the AI sift through it is a very wasteful approach.
It was adopted because trying to generate diffs with AI opens a whole new can of worms, but there's a very efficient approach in between: slice the files on the symbol level.
So if the AI only needs the declaration of foo() and the definition of bar(), the entire file can be collapsed like this:
Any AI-suggested changes are then easy to merge back (renamings are the only notable exception), so it works really fast.
I am currently working on an editor that combines this approach with the ability to step back-and-forth between the edits, and it works really well. I absolutely love the Cerebras platform (they have a free tier directly and pay-as-you-go offering via OpenRouter). It can get very annoying refactorings done in one or two seconds based on single-sentence prompts, and it usually costs about half a cent per refactoring in tokens. Also great for things like applying known algorithms to spread out data structures, where including all files would kill the context window, but pulling individual types works just fine with a fraction of tokens.
this works if your code is exceptionally well composed. anything less can lead to looney tunes levels of goofiness in behavior, especially if there’s as little as one or two lines of crucial context elsewhere in the file.
This approach saves tokens theoretically, but i find it can lead to wastefulness as it tries to figure out why things aren’t working when loading the full file would have solved the problem in a single step.
It greatly depends on the type of work you are trying to delegate to the AI. If you ask it to add one entire feature at a time, file level could work better. But the time and costs go up very fast, and it's harder to review.
What works for me (adding features to huge interconnected projects), is think what classes, algorithms and interfaces I want to add, and then give very brief prompts like "split class into abstract base + child like this" and "add another child supporting x,y and z".
So, I still make all the key decisions myself, but I get to skip typing the most annoying and repetitive parts. Also, the code don't look much different from what I could have written by hand, just gets done about 5x faster.
Yep and it collapses in the enterprise. The code you’re referencing might well be from some niche vendor’s bloated library with multiple incoherent abstractions, etc. Context is necessarily big
Ironically, that's how I got the whole idea of symbol-level edits. I was working on project like that, and realized that a lot of work is actually fairly small edits. But to do one right, you need to you need to look through a bunch of classes, abstraction layers, and similar implementations, and then keep in your head how to get an instance of X from a pointer to Y, etc. Very annoying repetitive work.
I tried copy-pasting all the relevant parts into ChatGPT and gave it instructions like "add support for X to Y, similar to Z", and it got it pretty well each time. The bottleneck was really pasting things into the context window, and merging the changes back. So, I made a GUI that automated it - showed links on top of functions/classes to quickly attach them into the context window, either as just declarations, or as editable chunks.
That worked faster, but navigating to definitions and manually clicking on top of them still looked like an unnecessary step. But if you asked the model "hey, don't follow these instructions yet, just tell me which symbols you need to complete them", it would give reasonable machine-readable results. And then it's easy to look them up on the symbol level, and do the actual edit with them.
It doesn't do magic, but takes most of the effort out of getting the first draft of the edit, than you can then verify, tweak, and step through in a debugger.
Totally agree with your view on the symbolic context injection. Is this how things are done with code/dev AI right now? Like if you consider the state of the art.
There's a pretty good sweet spot in between vibe coding and manual coding.
You still think out all the classes, algorithms, complexities in your head, but then instead of writing code by hand, use short prompts like "encapsulate X and Y in a nested class + create a dictionary where key is A+B".
This saves a ton of repetitive manual work, while the results are pretty indistinguishable from doing all the legwork yourself.
I am building a list of examples with exact prompts and token counts here [0]. The list is far from being complete, but gives the overall idea.
I'm still finding the right sweet spot personally. I would love to only think in architecture, features and interfaces and algorithms, and leave the class and function design fully to the LLM. At this point this almost works, but requires handholding and some retroactive cleanup. I still do it because it's necessary, but I grow increasingly tired of having to think to close to the code level, as I see it more and more as an implementation detail. (in before the code maximalists get in my case).
Most of the AI hiccups come from the sequential nature of generating responses. It gets to a spot where adhering to code structure means token X, and fulfilling some common sense requirement means token Y, so it picks X and the rest of the reply is screwed.
You can get way better results with incremental refinement. Refine brief prompt into detailed description. Refine description into requirements. Refine requirements into specific steps. Refine steps into modified code.
I am currently experimenting with several GUI options for this workflow. Feel free to reach out to me if you want to try it out.
Cool. The part that is released already is described here [0]. You can point at some code and give brief instructions, and it will ask the model to expand them, giving several options.
E.g. if you point at a vector class and just "Distance()" for prompt, it will make assumptions like "you want to add a function calculating distance from (0,0)", "function calculating distance between 2 vectors", etc. It runs pretty fast with models like LLaMA, so you can get small routine edits done much faster than by hand.
The part I am currently experimenting with are one-click commands like "expand" or "I don't like the selected part. Give me other options for it". I think, I'll get it to a mostly usable state around Monday. Feel free to shoot an email to the address in the contact page and I'll send you a link to the experimental build.
I use hierarchical trees of guidelines (just markdown sections) that are attached to every prompt. It's somewhat wasteful in terms of token, but works well. If AI is not creating a new wrapper, it will just ignore the "instructions for creating new wrappers" section.
It works with low-level C/C++ just fine as long as you rigorously include all relevant definitions in the context window, provide non-obvious context (like the lifecycle of some various objects) and keep your prompts focused.
Things like "apply this known algorithm to that project-specific data structure" work really well and save plenty of time. Things that require a gut feeling for how things are organized in memory don't work unless you are willing to babysit the model.
I don't see it as anything bad. LLMs are a interesting, alright. They can very quickly do a lot of mind-numbing work that used to be done by hand, and then they can stumble and produce total nonsense that no human being would even consider writing. And then you tweak the prompt in a weird way, and you're suddenly back in business.
To me, it feels like studying a new physical phenomenon. Like when Nicola Tesla was playing around with coils and wires, eventually loading to the creation of an entire industry.
Except, with LLMs, you don't need multi-million dollar equipment to play around with models. You can get pretty cool stuff done with a regular GPU, and even cooler if you use cloud.
I would say, if you are not spending some spare time fiddling around with LLMs trying to get them do some of the work you would otherwise do by hand, you are missing out.
So, there will be always business for someone to host one in a data center, and sell the tokens at a markup, that will still be cheaper than buying something you will only use 15% of the time.
I don't think it's currently possible to calculate a reasonable valuation for an AI company using conventional techniques. It is too new, it is still rapidly evolving, nobody knows how it will reshape the industry. It's more of a poker game where the underwriters are betting on how much the investors are willing to throw into a perceived once-in-20-years oportunity.
reply