Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
This absolutely does not pass any smell test, by your own admission you included the test files "by accident", could only barely pass the minimum of SQL for the benchmarks, included environment variables for "internal benchmarking" and a preview version you cannot immediately release because it might contain sensitive data? What kind of sensitive data are you putting in a Postgres slop fork in such a way that you cannot immediately release it, besides obvious admissions of cheating? This all just reeks.
> This absolutely does not pass any smell test, by your own admission you included the test files "by accident"
If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
I mean, if the now-released open source variant performs roughly the same, and has no weird behavior in it… the matter is resolved, no?
They’ve already submitted the best possible corrective step for any sus behavior they previously did.
It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
39% of people think it does more harm than good, compared to 9% who think it does more good than harm. Now 52% are equivocal, but as you can see, dislike of AI is much more common than like of AI.
I agree with "alot", but i doubt most "hate". Hate is a strong word/emotion that gets thrown around easily for sensationalist or propaganda purposes. Is this an "appeal to hate" argument?
Most people have more going on in their lives than to have chance to hate abstract things.
For most people AI are search results in "human" speech presentation. Or slightly more sophisticated customer service bot. Or the latest form of personalized clipart generator for their group chats.
For most people experiencing AI is not (yet) beyond the experience clash of going from nokia 3310 to a multicamera, ig/fb/doomscroll filled iPhone. Are they seeing much more "slop" in their doomscroll than in 2021? I'd argue not. Social media has been filled with low quality engagement triggering slop way before chatgpt.
Well, poll does not distinguish between "I hate AI slop" and "I hate any use of AI"
While I hate nightmarish ai slop pictures some restaurants use and some very ugly banners some street vendors use, I recognize some some AI generated assets are pretty neat.
Same as when I got annoyed that library I wanted to use turns out does not check any of the boxes it claims to check, because its pure untested slop. I would have no issues with it being AI if it worked because I used it to make my own versio of that library that in fact works as advertised.
> While I hate nightmarish ai slop pictures some restaurants use and some very ugly banners some street vendors use, I recognize some some AI generated assets are pretty neat.
So you don't think it does more harm than good, and think something else instead.
> Well, poll does not distinguish between "I hate AI slop" and "I hate any use of AI"
Isn't the criticism rather that the poll distinguishes between you and the people polled?
> So you don't think it does more harm than good, and think something else instead.
I don't think it does more harm than good. Maybe I'm biased because it's been very good for me and the people around me.
> Isn't the criticism rather that the poll distinguishes between you and the people polled?
No. The argument being made in this thread is that people hate AI slop (people just hate slop), rather than AI in general. This poll was linked as evidence that people hate AI. My argument is that the poll is flawed and does nothing to counter the point being made in this thread about slop vs. AI.
As for Steam...ever since they opened up who can publish and how publishing works, there been _a lot_ of slop pre-AI. I'd rather have more interesting and more controvesial filters. People will get surprised how few recent games aren't going to be filtered by "No AI"
Most people only see the slop. They don't have informed opinions. 1 in 5 American adults is functionally illiterate, and they are supposed to have a valid opinion about the usefulness of AI in contexts they can't see?
Most people hate what they (incorrectly) think AI is.
Somehow I feel like you don't bring this up whenever the general public gets polled and agrees on some other topic with your opinion.
Not even bothering to fight on the original topic and instead retreating to meta-analysis titled "Everyone is stupid anyway, I am the one who's truly informed" doesn't make for a very convincing argument.
Oh, well good thing you feel and assume how I think and debate. Perhaps your invented picture of me is not accurate?
You might not like it, but that doesn't make my argument inaccurate. Every discussion I've had about the topic, anti-AI folks are willing to acknowledge exactly 0 benefits of AI. That is just dumb, because benefits do exist, and it prevents a rational discussion and analysis of costs and benefits.
When a side in a debate cannot or will not acknowledge the strengths of the other side's position, they aren't having an informed argument; they are ranting. (This is also the chief issue with contemporary electoral politics globally, imo. People arguing from principles rather than facts.)
Finally, I don't claim to be the only one who is informed; where did I say that? I didn't even claim that anti-ai people are stupid. I do claim that most anti-ai people are uninformed and this is because in my experience, they are unwilling to acknowledge any benefit of AI. Further, I propose a mechanism for their uninformed opinions: because slop is highly visible, whereas benefits are mostly invisible. As examples, take drug discovery and deeper understanding of maths: do you honestly expect a population where 1 in 5 adults is functionally illiterate and the literacy and numeracy scores of our youth continue to plummet will value those things that are literally not things that can be seen (and rely on very advanced degrees to understand), versus the prevalent and highly visible slop that exists?
Telling people to ignore the evidence of their eyes is a hard sell. They see a ton of AI slop everywhere and they think that AI is harmful because it turned every site and every platform and every type of human expression they’ve experienced into endless piles of AI slop. Can you blame them? Are they even wrong?
>Telling people to ignore the evidence of their eyes is a hard sell.
Yes, I agree. It took science to prove that life doesn't arise from rotting meat. But the fact that populations (even in developed countries) are rapidly getting less and less educated indicates that people are less and less likely to have the critical thinking skills to understand when their eyes are not adequate instruments of observation.
Also AI slop exists, certainly; people aren't wrong to hate AI slop, but you have to agree that slop isn't the entirety of AI.
thats funny, because you are moving the goalposts in the argument i made, which is that people who say they hate AI don't know what AI is and think it is only slop, yet you readily agree that AI is more than slop.
I've even seen videos that were in English, but mislabeled, so they got auto-dubbed back to English, replacing the real voices with a monotonous synthetic voice.
I appreciate the auto translated captions sometimes as they let you get the gist of what's going on, but auto dubbing is pretty much never desirable
Because they selectively choose which benchmarks to highlight and which competing models to compare against; they don't always compare against the SOTA or the Pareto frontier of performance/price, a lot of times you'll see them compare against the last generation of competing models, or the like, so that they look better if you don't carefully consider what they're comparing against.
For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.
Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.
And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.
Being pedantic - LLMs are deterministic software, that is chaotic and typically deployed with a PRNG seed. An LLM ran using the same seed will produce the same output consistently (theoretically - in practice this isn’t always the case because of changing factors wrt the execution environment given how sensitive/chaotic models are).
In many cases they are not deterministic even with a deterministic seed because many runtimes embrace races that interact with floating point rounding to improve latency.
in practice, as actually used, LLMs are nondeterministic.
It is possible, if you control the execution environment sufficiently, and are careful not to use certain kinds of parallelism which can lead to nondeterminism, and you set the seed deterministically and ensure that the prompts are byte identical, you can operate them deterministically.
But no one actually does all of that. Almost everyone uses cloud hosted APIs where the provider could change any number of variables at any time. Everyone wants faster cheaper inference so uses inference engines which may trade off a certain amount of determinism for more speed. Different versions of harnesses and chat templates and benchmark software may introduce small differences in text.
And even though the cloud providers have in the past offered seed and system fingerprints to try to give people a way to control or detect non-determinism (see https://developers.openai.com/cookbook/examples/reproducible...), that has all been deprecated and removed in the newer APIs.
So yeah, while in theory it's possible that if you control the whole stack you can get deterministic results, that's not how anyone is using it, and there is no way for third party benchmarks to ensure deterministic results of hosted models, the latest APIs don't even support it at all.
This whole "well, actually, LLMs are deterministic if you hold them right" is a red herring. In any actual real-world use, they are basically always non-deterministic.
And of course, there are reasons that providers have dropped these attempts at providing deterministic results; because the results are chaotic, perturbed by small changes in input, it's almost never useful to try to make them act deterministic or detect when non-determinism has happend. Instead, it's better to think about them non-deterministically, and look at the distribution of results, rather than just considering a single pass or fail on a single test to be meaningful.
With the old GPT-3 api in the olden days, you could set the seed and get deterministic outputs! Interesting results could be drawn from it, as for instance if you use the prompts "a blue bird"/"a blue bird in the sky" the blue bird would have the same look in both with similar seeds.
Fair. I wasn’t advocating for “holding them right”, I feel it’s an important nuance to keep in mind when conceptualizing them, i.e. it’s procedural, determistic, yet both inherently and in practice extremely chaotic and therefore not practically deterministic software. So, in essence - software.
Why would you think that the author was speaking for anyone but themselves?
When I say "juggling is fun", I'm talking about my own experience. I'm not saying that everyone else should drop their hobbies and take up juggling.
Why should the author have to couch the headline of their article with this kind of defensive language that you suggest? The very first word of the article is "I", it is clear that the author is talking about their own experience.
They did come back for a while after the original cease and desist, but they've shut off again "due to a new development in the ongoing legal proceedings" a few days ago.
There are LLMs that support audio input, similar to those with vision support.
From my testing of open weights LLMs with audio support, they basically are only trained to recognize audio as an alternative to text input, they treat audio as basically equivalent to a transcript, and can't recognize or distinguish things like music, accents, background sounds, etc.
So they're only really good for transcribing or summarizing or using audio input in place of text input for prompts, but not anything that requires distinguishing any information about the audio that would not be present in a transcript.
It can be tempting to try to use an LLM for a variety of tasks; kind of the whole thing about an LLM is that you don't have to do a separate complex training run for every task, but can just provide instructions in natural language. But it only works as far as what the training data covers, if the training basically always treated audio and a text transcript as equivalent, the model has nothing causing it to learn other relevant features of the audio. If there's enough bird call identification in the training data of an LLM, it might be able to do that, but I think multimodal training data tends to be much more limited than the text training corpus
Have you had any luck fine tuning one with musical data for classification or music-aware QA? I've been hacking on https://trebel.la/ which I would like to be a music practice companion, and the biggest missing feature is actually useful audio-based feedback pipeline.
My current approach, not yet validated, is trying to generate training data from masterclass recordings on Youtube, and then fine tuning MOSS-audio on a bunch of those. But I'm interested if there are better models, or large training sets I don't know about.
I'm not selling anything; I'm developing it live and there is a landing page, but there's no payment hooked up since like you said, it's not functional yet.
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
reply