This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
A DGX Spark is currently less than 10% of the median US household income (~80,000 USD/year), but that's still a big expense for the middle class. Until July or so, it was under 10% of the median personal income (~47,000 USD/year).
I personally put a fairly high degree of value in the 'genuine' investigative journalist you're referring to.
The more recent spate of sensationalist journalists however are less remarkable. The ones that parrot headlines from other places, or worse paraphrase to make them get more views.
I absolutely agree that those who do not take their jobs seriously are not praiseworthy.
My pushback stems from what I perceive as a lopsided amount of ire directed at "journalists" without nuance.
I don't think most people make any sort of distinction when they voice this ire, as we see above with the commenter I initially responded to decrying "journalism" as the "lowest of vocations" and claiming it has never had any value.
Clickbait/yellow journalism has always been the cost of having a free press. We should shame those people when we see them, but should not throw the baby out with the bathwater.
My solution is buying the cheapest stuff (about 8aud/5euro?) a bottle that I reckon tastes great, knowing full well I enjoy the taste and that's all that matters to me!
I would say I used to believe how the models would scale, now I'm a bit more skeptical. I would be incredibly surprised if oai, anthropic, etc hadn't at least TRIED this yet. They have the compute to do it. They're either sitting on it and saving it for a rainy day, or there is something wrong with it.
We know nothing about their internal models that, for instance, produced revolutionary results in mathematics. The public no longer gets to see what’s at the frontier, and we may never see much of it. Also, the cost to train these is immense. China literally does not have enough chips in circulation to compete. I’m not sure where US lab capabilities tap out but I suspect they are at least partly bound by GPU production as well at this point.
We know that if this immense cost had in fact been spent, and the internal models really were that revolutionary, these companies would be selling access to it for cash instead of taking on mountains of debt.
I agree they would not shut up about it if they had something to reveal, so we probably have a semi-accurate vision of the capabilities if not the implementation details.
I honestly don't get why companies need to train on prompts any more? Surely they've got PLENTY of training material, and the advancements these days come from architectural improvements?
Not at all. As companies refine their models to perform better at specific tasks, they need more data to train on that specific task. I know first hand that the frontier labs are reaching out to all kinds of companies asking to license their software to run so they can train on it. The stated reason is not to reproduce that tool, but to get better at operating it, though history tells us that reproduction will naturally follow even if unintended.
We have talked to many customers who specifically wonder if our software could learn to perform better for them by learning from their interactions - they are suggesting we train on their work! This tuned model would only be allowed within their enterprise, of course. It's understood that the models know how to do this generally, but every situation is different, and results could be better if taught how these specific situations work.
I just wanted to say thank you for posting this. It was a wonderful break from the norm, and I've enjoyed reading everyone's comments. I haven't enjoyed anything here or in general in a while.
With luna's pricing, is the power consumed not likely to more than the cost of API calls?
At 50tps on a 3090 for qwen 28b or similar, that's about 200k tokens/hour. Which is what...10c for Luna (balanced input vs output). 1h of power at 30c/kwh is 30c? So you get a worse model, for more cost?
I pay about $0.14/kwh and don't need it all the time. API calls might be cheaper. I develop in a way where I break things out where well defined tasks can be handed to a local model.
I think the opus5.5 vs fable5.1 situation has been my tipping point where I completely ignore these benchmarks now. Fable is just better than Opus5.5 in all cases I've ever used both and compared. I don't care what the benchmarks say, they're just not reflective of my needs, so they're not relevant to me - other people may have different experience.
reply