He won't be punished by the legal system.

Possibly. But even if not, how many AI model developers do we need? We will likely need a nonzero number, but it could be very few compared to the total number of software developers we have today.

Yeah I'm really impressed by what Opus 5.5 can manage.

I feel like human code is on a wide quality spectrum. The best human code is way better than what Opus 5.5 can do. But the average code, I'm not so sure.

I have only very rarely in my career encountered code worse than what an LLM produces. It happens, but your typical programmer is still quite a bit better than an LLM.

Could definitely be I've had the misfortune of working with below-average programmers (and inheriting their codebases).

The way I see it many HOAs are the opposite of privatization, at least when they cannot be opted out of. They control land, impose taxes (sorry, "fees"), have elected leadership, provide services, make rules that they have the power to enforce, and the only way to leave them is to move out of their territory.

In my eyes this is essentially the same thing as a form of local government, just on a smaller scale.

The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.

It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.

Where I live, even when government does make demands, as in the case of foster parenting, they still allow two boys or two girls to share a room.

The insane part is placing the burden on AI labs to advance mathematical understanding. Why should this be their responsibility?

If they want to simply generate theorems and proofs and post them on the internet, why not? Does doing that really inflict "substantial negative externalities ... on the mathematical community"?

Also, people would get mad at them (rightly) if they sat on a proof of a major unsolved problem for months while bringing in human mathematicians to convert it into a more human-friendly form

(They definitely should spend SOME time working with human mathematicians before putting out a press release, but that's exactly what they've done every time)

If math is a "burden" on AI labs, why do they willingly spend tens of millions of dollars for a purely theoretical and unphysical result? What's the ROI for that when the frontier labs are already notoriously cash-strapped? Their motives pervade the entire discussion and cannot be ignored here.

Presumably some combination of marketing existing models and training/improving future models.

My point was about what is and isn't the responsibility of AI labs. Why should they be required to invest more (or fewer) resources into math than they otherwise want to?

(If you ask me, safety and security are what they need to invest more into.)

Simple. Because they are benefiting from output of the math community, in fact they were required for creating these models in the first place.

...Could you not ask this of anyone who works on pure math problems? What on Earth are you talking about??

"Academics are paid paltry salaries, university departments are cash-strapped, why are they studying these abstruse math problems instead of building rockets?"

Successfully advancing fields of math and science is a good thing. Why would they not do it. Why would they not want to prove their increasingly intelligent computers are able to solve unsolved, complex math problems humans have failed to solve for centuries and then release the solutions to the public. What do you want them to do, not try to solve them?

They are not advancing the field. At least they didn't in the case of the N-S debacle. The model most likely used the conversation text of a mathematician which had discovered and was pursuing a new avenue towards the goal, then pumped millions upon millions to hurry up ahead to an opaque solution which leaves no-one the wiser, beyond just settling a theoretical question. The only impact on science is to deflate the potential minds who might have been motivated to join the effort.

Because it's faster.

I've done this on different topics - I know the answer is in a particular eBook/PDF/document, but for whatever reason it's not trivial to look it up. The model can do it a lot more quickly than I can, and then I can still verify the accuracy.

Depends on the harness too. The MS copilot 365 chat interface happens to parse PDF and docx files before handing them to the model. Absolutely fantastic if your PDF is 1300 pages of concatenated reports and you want to find a single detail in it that you can't easily ctrl+f for.

I wonder how the ratios would change for papers at different parts of the review process. For what fraction of published papers are the authors unable to answer basic questions about them?

That sounds incredibly frustrating. It's bad enough in code reviews, but is that approach to work being tolerated in meetings these days?