I like how Tyler Cowen puts it: "This is the worst they will ever be".
And some people are still stuck on what AI was capable of when GPT-3.5 was SOTA. They don't realize how much they've improved in ~4 years. Even if the rate of improvement slows down, AI will be tremendously more capable in another ~4 years.
Not my experience, an I'm a heavy daily user of frontier models for software development. Have they improved at all in the past ~2 years? Not really, at least not noticeably. We've improved some of the tooling around it, developed some mitigations for the many systemic weaknesses, and wrapped some things in a for loop to iterate faster. The worst they'll ever be? Maybe, but maybe also almost as good as they'll ever be, looking at the sharply diminishing returns. Reactions to every new model seem to always be roughly evenly split between it being better or worse.
And what do we have to show for it? We can localise some bug causes faster, and it takes some of the mindless boilerplate work off our hands. Which affects maybe 10-20% of our actual job, and even there occasionally eliminates the gains again right away because you have to babysit or manually redo it, without always knowing when and where that might happen. That's not what I'd call a revolutionary increase in efficiency.
And this isn't just my impression, it's what I hear from pretty much all of my professional peers, which includes people waaay more experienced and smarter than me.
This is a surprising opinion to me! Subjectively, I very strongly feel that models improved significantly over the last two years, particularly around the time of Opus 4.6 and now Sol 5.6 and such. It's made a huge difference to how much autonomy I can give them.
If you trust benchmarks (which is debateable) then the rate of improvement is speeding up -- definitely not diminishing returns.
And some people are still stuck on what AI was capable of when GPT-3.5 was SOTA. They don't realize how much they've improved in ~4 years. Even if the rate of improvement slows down, AI will be tremendously more capable in another ~4 years.