There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
I wonder how much different that is from humans, even professional comedians. You don't usually get to see all the jokes that comedians try that don't work, unless you're an avid attendee at comedy clubs.
For comedians, perhaps. But there are plenty of humans who aren’t very good at comedy.
In these LLM comparisons, people tend to compare them to the best humans, not average or below average. It’s a kind of motivated reasoning, with the goal of deprecating the abilities of LLMs.
Well, you did ask for a comparison to professionals.
But I think humor is pretty easy to rate for a human. Did it make you laugh? Yes? Then it might be a good one. This simple human filter radically improves your efficiency as a joke validator, leaving any LLM far behind.
For pictures I'm not sure if humans are much better. For music, yes, after a few listens (recognizing a hit usually takes time). For games, definitely yes.
The hosted GMI cloud model on openrouter does not invoke confidence. Ignoring 131k context length limit the tool call failure rate is not acceptable.
When asked to call 'submit_html' tool for our bench it wrote the whole page as plain reply text, then called submit_html with "This is the complete submit trigger." 4/8 tools failed.
Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
I agree. Flash 3.8 is not good and actually quite expensive, given its tendency to overthink everything. We've been transitioning lots of workloads that were formerly on Flash to Haiku 5.5. Great model.
Even with gemini 2.5 or so i remember the models used to just simply hang in production and randomly used to collate tokes and give in the output stream vs giving one by one which was weird.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
In our own benchmark we found that K3.1 pro UI designs are very similar to OpenAI's designs.
K3 for reference had 0.43 similarity to OpenAI's models, now it's as similar as how OpenAI's models are to each other at 0.58, much higher than other cross-lab similarities.
Which benchmark? Would love to see it & any datasets.
It’s super tricky to tease out whether the similarities point towards distillation. A lot of the aesthetic similarities are naturally emergent from the training data.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
This statement is also incorrect
reply