"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
This is mostly a self made and kept problem. I also worked in codebases where the current developer(s) didn't know why things were the way they were and would take days to dig into why some part wasn't working - but the issue was, instead of taking the time to write tests and document, even if only when doing these point in time fixes, they just did the ad-hoc thing and solved whatever issue there was at the time without any worry whatsoever.
At some point I was tasked with improving the queries/retrieval times and started converting parts of the code and not only was it possible to write tests that would convey the actual meaning of the code and crystallise it but I started finding bugs in the implementation that no-one realized were even bugs because they no longer knew what it should do when queries grew to certain complexity and the search space was in the hundreds of millions of rows that thinking through them just with napkin calcs wasn't cutting it (specially when the bugs were subtle and the results still allowed the program to work "mostly" correctly).
And this holds true for most of the Internet. There is no one person who truly understands how it all works: DNS, routing, and all the other things we rely on to get our systems to work.
Network engineers and TCP/IP stack implementers have pretty good understanding of how the Internet works.
But I have see quite terrible finance software, patches over patches hold together by a duct tape. Also finance software has to implement large complexity of tax law, which is very large and always changing. It also doesn't help if the company developing finance software sees software development as a cost center and wants always to push the costs down (especially short term costs).
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
If anthropic is making as many PRs as stated in articles, then rolling back is unlikely an option. More likely it'd be a new commit to fix it, and with a feature flag to preserve the "wrong" behavior (even if wrong, since people will probably depend on that function since its original behaviour).
Those sorts of changes are really rough to manage. Maybe still manageable by LLMs but we'll find out.
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
Is the Hugging Face incident from July really a problem? Or Australian healthcare portal breach? Who let their guard down unable to detect or roll back the change for days on end?
As I said at the end of my comment, unless you're an AI researcher pushing the bound of what LLMs can do. I don't think most of us are running ten million dollar AI swarms with unreleased models that are escaping their sandboxes. That's research, not software engineering, and this discussion is about whether or not software engineers can reasonably expect to understand their LLM generated code.
Also, for what it's worth, LLM research is hardly the first scientific research to have accidents with negative external consequences.
Frontier labs are not the first to socialize externalities and privatize the gains. Not even the first to shift the responsibility on to some non-entity (agents, corporations). I think you have a very reductionist view of software. Software is used to design the buildings you inhabit, bridges you cross, vehicles you drive, medical devices you wear...
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
Here is an explicit example of the additional lack of coherence due to LLMs in the Anthropic Claude Code public repo itself: the human maintainer trusts LLM's incorrect understanding of the current state of code in the repo.
> "Issues here are auto-closed by a bot when they've had no activity for a while and are missing information needed to reproduce (usually a version and steps, or a /feedback ID). Nothing is triaged solely by counting"
Therefore, I wonder whether the internal review process described in the Anthropic's blog for internal repos also results in such issues, and whether Anthropic has a internal way to measure and mitigate that. The blog post says:
> "When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today."
The Claude Code public repo problem shows that LLMs don't always "reflect the code as it is today", and can be a source of additional misunderstanding.
Exactly this. Using an LLM to understand the original LLM slop is just another chance to get it wrong 20% of the time. These things just aren't reliable.
Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.
What you are saying is leaning on 2 premises that seem to not hold water
1. Code written by people is always more readable and maintainable than LLM code
2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
False assumptions.
The human doesn't have to always be more readable, to be usually more readable. And vice versa. LLMs given the opportunity to generate a "feature" almost always generates more complexity than a software engineer would develop.
I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.
"I've seen bad code in the past" does not address the fact that human code by an experienced software developer is almost always less complex and more clear, with less repetition, dead code paths, and odd architectural choices than what an llm generates. If you don't recognize this, then you either haven't been reading the code or are not an experienced software developer. 80% success on a benchmark means they fail 20% of the time in the exact domain they were trained on.
I am an experienced software dev, and I am telling you this as clearly as I can: human generated code is oftentimes buggy, indecipherable slop.
You need to put 80% in context. There's a reason why FizzBuzz lasted as long as it did - lots of developers, even senior devs, are terrible at writing code, subjected to absurd time constraints, or required to color inside the lines of a leaky abstraction. Sometimes all three!
I've seen many situations at my current job where asking an LLM to do a refactoring made things much better - but if I had to do it by hand I'd just give up and move on to more essential work.
I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.
Thanks for that. Dune used to be my favourite sci-fi, but, The Expanse maybe trumps that now. I'll definitely check these out once I finish The Long Earth :)
It's not clear to me: do we need to both apply the patch _and_ check if our themes are vulnerable, or just apply the patch?
Edit: looking at the patch itself it looks like it fixes the root cause and so it shouldn't matter what themes are doing themselves. But possibly I'm reading it incorrectly.
I'm sure this is a basic question I should already understand, but, what is actually in it for the bot farm? They spend your money (annoying), but, does Google actually pay them somehow? I don't get what they are "farming"
Yes AdSense pays 3rd party publishers locating your ads on their site. They fake their site click through rate with bots, and get paid as if they had real users clicking the ads
I've been helping to run a meetup in Christchurch, New Zealand: https://christchurch.ruby.nz/ for about 10 years (maybe longer, I've lost track!). I _completed_ agree with the article that the best thing you can do to avoid burnout is to organise it with someone else (or better still, two or more others like we do).
The other thing which really worked for us: We were getting fed up trying to badger people to help give talks etc (or just doing it ourselves). Completely exhausting. Ended up saying to the community: "this is your meetup, we just help run it. If we don't get 11 speakers for the year we're going to shut it down". We asked in January, got 11 speakers for the year (Dec is social), and we've been doing it like that ever since. Works an absolute treat. You do need to be prepared for no one to step up, in which case, it's sad, but, shut it down. You're not the community, the community is the community!
I run the Python meetup in Thessaloniki, Greece, and it's exhausting trying to find speakers.I frequently skip months because nobody wants to present anything, and we aim to have two presentations on each meetup. I'm almost always the second presenter, and sometimes I'll do both presentations myself (or just skip that month's event).
It is exhausting, and people do always ask when the next one is, but my reply of "whenever you want to speak" doesn't seem to do much to get more speakers.
I haven't looked into it thoroughly yet, but, at some point it'll be worth looking at getting a static IP address and using Cloudflare tunnels to serve toyish projects from your home netwerk. Just need to work out how to firewall it properly from the rest of the home network.
Though quite possibly I'm missing something ;)
---------------
edit: as I was typing the index was updated. So, as far as I can tell only just now (18:20 UTC) was it available for updating!