Hacker Newsnew | past | comments | ask | show | jobs | submit | silveraxe93's commentslogin

It's not willfully ignored. You might not like the answer, but the position is very clear.

> It is simply unethical, unfair, and devoid of empathy for society to simply "march forward" and leave behind the millions of people it spent the last 60 years demanding more of.

We disagree. This is a luddite point that I spent my whole life disagreeing.

I think it's unethical for society to intentionally block a technology that increases productivity to protect the interests of a smaller class of professionals. I believed that when it was happening to blue-collar workers, and I'll keep believing it when it's me.


Luddism is often mistakenly looked down upon as “progress bad” and “screw technology, destroy machines”.

In reality[0], luddists were much more along the lines of “let’s be careful about progress” and “let’s negotiate that, while machines are used, humans are not fully superseded, keep earning livelihoods and feeding their families”.

If said negotiations fail, it does become “destroy machines”, but the way it tends to be taught to us, unsurprisingly, skips a crucial detail.

[0] https://en.wikipedia.org/wiki/Luddites#Peak_activity:_1811%E...


> I think it's unethical for society to intentionally block a technology that increases productivity to protect the interests of a smaller class of professionals.

And that's how we all wound up with Whispering Earrings. https://croissanthology.com/earring

Maybe you genuinely want one -- I don't think many people do.


Death of the author and all, but Scott Alexander has long insisted that the Whispering Earring is not about dangerous technology: https://web.archive.org/web/20121007235422/http://squid314.l...

We’re not talking about generic “dangerous technology.” We’re specifically talking about a policy that says the government can’t ban productivity-increasing technology.

Great short story! Thanks.

Yes, the point of the article is that he'll stop uploading _unless_ there's an opt out.

> I will not upload anything to NEW unless there is a way for me to opt out of this. I would prefer if the default was "no LLM review" and that developers have to explicitly opt-in to receive an auto-generated review.


That's not false. That's why he literally asked to be able to opt out.

> I will not upload anything to NEW unless there is a way for me to opt out of this. I would prefer if the default was "no LLM review" and that developers have to explicitly opt-in to receive an auto-generated review.

But as the system stands he only has those 2 options. Which is why he will not use the system until they add a 3rd one.


I must have misread that, thanks. This seems like an entirely resolvable problem.


I don't believe you. Unless you stretch the definition of 'frontier models' or had a prompt that's literally asking for that.

I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.


I believe them. I had Gemini-3.1 Pro do essentially this seven months ago - specifically a loop of writing "Bye! Done!" repeatedly until manually stopped. Presumably some sort of pathological behaviour where it wasn't ranking the stop token highly when it should have been... but damned if I know why... it was an innocent prompt (and it had done the thing I asked it to before failing to stop).


huh fair. Still very surprising. I have a _very_ unfavourable view of Google models, so I can kinda believe 3.1 pro could loop.

Still would never have predicted this. Even 2.5-flash was pretty good. Never saw anything like this happen.


It's really uncommon now, but it's still possible. I hit it two days ago with deepseek 4.1 flash, and I've had coworkers report that happening with Gemini.


quants still do this and the token vendors serve those up when demand is greater than supply

things have definitely improved significantly, however "never say never" is still good advice


It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.

Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.

The floor of software quality once it gets to review is permanently raised.


I really don't get the issue here. Is the rust borrow checker dystopian?

You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.


> You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

that's called the write/compile/check loop

but if suddenly it's ai, people freakout at having to do this?


It was never the skilled way of doing things.

Also an AI is not a compiler, stop saying that.


Compilers transform source code in one language to another target language.

Transformer models can transform natural language to another target language.

That’s the commonality. What’s the distinction you want to make?


Let me introduce you to the reproducible builds project: https://reproducible-builds.org/


What a childish way of saying “they’re not reproducible”.

Human output is not reproducible either. I take it your argument is that we should remove humans from the loop, then?

No? In that case, you have some reconciliation to do.


_Is_ AI generated code notoriously terrible at security?

My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.

Are we comparing to old models generating code? Or mythos-class models?


It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.

There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.


Its not entirely a prompt issue, but also what it was trained on.

Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.

Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.

My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.


Most software is written without giving a thought to security. Because security is a source of cost. That was the case before LLMs, I don't think it will change with LLMs.

Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.

As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.


I think that even worse is that AI might choose just to write something, and skip using a library that is well maintained


Or use an out of date library.

In our somewhat “developer assisted” AI development it planned to use a library. We looked the library up and the GitHub page indicated it was “no longer maintained use other library Z”. When told AI agreed and compared the original to the new library (helpful), but it seemed to heavily weight GitHub stars more than active development.


When a project talks about how many stars they have, I immediately mistrust them.

Usually download counters and stars are signals for VC, which indirectly serve also as negative signals to me.


Tools like SCA and SBOM are still needed to ensure your stuff isn't out of date. That's not exactly a problem an LLM solves.


This is where a human with a discerning taste and wide knowledge of the field should notice it during a review, or at least instruct their agents to check for that. If such a human is still employed in this capacity, of course.


that can sometimes be a plus considering how dependencies used by a lot of projects are themselves targets


it costs time until the new Opus7 is trained on hardened code.


Physician heal thyself?

Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?


I’ve found they won’t always comply with adversarial review.

No doubt because us unwashed masses aren’t blessed with the super special “security” ones.

We might be looking for exploits for bad reasons, I guess.


Depends where it is blind at.


Adversarial fixing. I like it


It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.


Yeah they are pretty bad at it still, latest models tried to ship a basic SQL injection flaw. Was also happy to build out an xss flawed module end-to-end. Definitely have to keep a close eye on it.

My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.

Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.


What models are people actually using to pump out code? That is the relevant comparison, not the absolute best in class models.


It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).

The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.


That study is from 2022, with at most a last update 2023? (if I read it right)

I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".

Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.

I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.


It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.

Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.

All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.


from your paper:

> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.

That model came out in March 2022, it's GPT-3.5 !!!

I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.

You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.


Hence why I said the studies here are quite old, and I'd like to see something more recent, though I guess it'll be some time before that's done.

I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.


I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.

AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.


I agree that we can’t compare older models to newer ones, but how much code was generated during those months/years when the models weren’t good at secure code? How much is still running publicly today? It’s still worth considering the impact to the overall ecosystem.


Luckily we have newer models going back and fixing these things.

In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.


like most software, do you assume people are driving lamborghinis, or the cheapest make and model they can find...


Exactly. You told the dog to 'sit' and it didn't listen to you.

It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.

Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.


I don't think they intentionally set it up to hack stuff with a prompt saying "hack this site".

I do think it's highly likely they knew this would happen with the lack of safeguards and number of instances of this stuff they were setting up, and that it's PR they want to make the models seem "powerful". Stochastic "unexpected" events they can advertise.

I suspect it was probably set up with the official internal goal of just trying a ton of arbitrary tasks that seem hard so that when any of them succeed they can publicise it and pretend the models do that routinely, but a "failure" where they hack stuff works just as well, if not better, for their goals.


They absolutely set up the agents to hack stuff with a prompt like "hack this site" -- they just imagined that their lazy half-measure precautions would prevent it from actually happening. They were defeated by a combination of bad luck, poor planning and tenacious agent ideation.


I don’t buy them being surprised. This perfectly plays into their pattern of using fear to make their models seem more powerful than they are. They obviously had the technical expertise… there isn’t a damn thing a bunch of randos on some HN thread knew about that model that they didn’t. It gave them an excuse to delay their IPO when their books seem like they’re going to be pretty shit compared to Anthropic. It helps them reposition themselves as being more safety-forward which the market is clearly more interested in. All that is to say they had motive out the ass, easily had the knowledge and capability to avoid the problem, knew better than anybody else what the models were capable of, set up the environment, gave it the prompt, and then did not even monitor the output.

Negligence is carelessness. Recklessness, is disregard for a known, substantial risk.

I absolutely believe this was recklessness.


That would be a completely insane thing to do. I suppose it's possible, but I really doubt it.

"Let's widely publicize a tort/crime that our computer systems did, and then cross our fingers that nobody ever sues us or does even the most basic investigation that would immediately uncover our criminal conspiracy."

In the insane corporate crimes you read about (maybe FTX, or the eBay stalking scandal), they were trying to cover things up, not heap public attention on it for months.


Huh? It's extremely common for businesses to decide that breaking the law is a cost of doing business and just do it because they figure they'll end up net positive from it, or even just to cash out in the short term.

Uber made no attempt to cover up that they were operating without licenses, and just ate it and fought it betting they'd get established before the law could catch up, and they won that bet, paying some fines and stuff but ultimately taking the market.

These LLMs are literally trained by these companies pirating literally every bit of media humanity has ever made, they made very little attempt to cover it up.

Of course they'd be willing to break the law for some PR? With a thin layer of plausible deniability "oh no, we didn't mean for it to hack stuff!" they know they'll get a slap on the wrists at worst, all while generating hype to prop up the AI bubble further by presenting the models as hypercapable.

The mindset is probably: either a) the models become capable of what we are claiming and so the companies become so huge and valuable the cost is irrelevant and we'll be to big to punish meaningfully, or b) it's a bubble and might as well push it up as big as it can go while I can make money, then by the time consequences come around I'll be long gone and who cares.


I'm not fond of analogies, but in this case I agree.

There is a clear difference between OpenAI intending to hack something vs. OpenAI being negligent in the creation/instructions of the agent. But the latter still leaves OpenAI liable for the agent's actions and calling it a "rogue agent" doesn't avoid that.

Moreover, with a dog, we don't rely on training/alignment to prevent bad outcomes. We rely on physical restraints like leashes and muzzles. The AI's tools to access the outside world should have been restricted. Perhaps instead of giving the AI arbitrary HTTP access, it should be given semantic operations with restricted URLs, etc.


If an agent has cheated once to achieve the desired outcome, and the trace is used to train further models (RLVR), then OpenAI is effectively telling the agent to cheat/hack from that traces' inclusion in the training set.

So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.


OpenAI trained "the dog". They created the system that would "hack" if given instructions and they gave those instructions


RL systems doing unexpected things isn't exactly new, so not sure that is then "rogue" if it were a property of the thing itself and not an active decision.


Yes! Exactly!

Sorry don't wanna seem like I'm raging at you, take this as a shout towards the void. But this exact fucking thing is what the less wrong / Yudkowsky crowd has been warning for years.

Now you say 'so not sure that is then "rogue" if it were a property of the thing itself and not an active decision'

Like that's semantics. It really doesn't matter. What matters is that we have a paperclipper in our hands. The _only_ difference is that it's not superintelligent. But if it was, then it you'll get decomposed into component atoms while saying 'aha! But it doesn't _actually_ want to kill you, stop anthopomorphising it'.

I get not being worried about x-risk because someone just doesn't believe in super-capable AI. That's totally fine.

But that's not what people argue. People have been saying for a _long_ time that AIs are aligned. And now when this happens it's either 'It was marketing, they intended to let it loose' (WTF! The levels of motivated reasoning to believe that are unreal) or 'yeah I knew that', which fair. I also thought that! But _that_ is why I'm fucking worried.


Practically, that's exactly what Chavez did. Where did that lead?


To be fair, oil nationalization and nationalization of incredibly advanced technology companies are not the same thing. See Norway as for what's possible if you do resource nationalization correctly.

But of course they do understand that and the primary nationalized the resources, and have a model where the nationalized company bids for that resource against private companies as well.


Not sure, you cited one of the most popular surnames in latin america.


> However, the declaration argues math is more than a machine for producing correct answers.

There might be more to maths than that, but that is definitely the most important part. I love science funding. But not because it's a jobs program for nerds.


Culturally, mathematics is a jobs program for nerds. The field very explicitly takes pride in working on problems that have no obvious applications, and most practitioners are funded publicly or supported by private endowments, with zero pressure to deliver specific results.

Of course, this produces useful results every now and then, but it's not like we pursued ruthless efficiency / maximum rate of knowledge advancement before. We just let them do their thing, essentially treating them as artists and letting them pursue the craft for its own sake. If we weren't interested in maximum throughput before, why is that an objective now?


Hardy would agree with the viewpoint that you espouse but it would be pushed back against by Arnol'd, Poincare, Gauss, Von Neumann, and even Grothendiek: Arnol'd and Poincare were vituperatively against the division between "pure" and "applied" mathematics; they considered mathematics and physics interchangeable, and Arnol'd lamented that the field had lost a large amount of funding/prestige/relevance due to groups like the Bourbaki that took a purely aesthetic view; Gauss had a critical view of problems like Fermat's last theorem (he felt that you could construct infinitely many such problems, and felt that attempting to prove it was a generally useless endeavor), along with outright calling pure mathematics worthless; but while Von Neumann and Grothendiek were more moderate, both were critical of the field losing motivation/quality as it strayed away from empirical science into—quoting Von Neumann—"abstract inbreeding".

Arnold's polemics are perhaps the most infamous and easily found online (see "On Teaching Mathematics"), but the written opinions of Poincare et seq. are also easy to find. Even today the vast majority of research funding for mathematics, at least in the United States, is dolled out for highly applied fields like partial differential equations. The field does not even close to unanimously (contemporarily or historically) "explicitly take pride" in working on problems that have no obvious application, or being a "jobs program for nerds": the notion of such "pure" or "nonapplied" mathematics is at the very least a highly fractious and controversial subject, with a number of big names taking opposing viewpoints (often vehemently).

I think your picture of the field is over-represented on the internet, much like the fixation on certain niche fields: Category Theory, Homotopy Type Theory or, worst of all, outright dubious fields like Geometric Algebra; fields with a large number of online promoters, but with much less funding and relevance in the actual academic space. Of course there are reputable people with PHDs that feel this way,—but I can only imagine that there's a legion of tyros, pop math consumers, and undergraduate students who disproportionately promote this viewpoint.


What’s wrong with geometric algebra? The same could’ve been said about linear algebra 100 years ago.


> We just let them do their thing, essentially treating them as artists and letting them pursue the craft for its own sake.

I think we generally did that because that seemed to be the best known process for maximizing the quantity of useful mathematics that they occasionally stumble upon.

It's not like we treat math as a charity project for eccentrics who like blackboards. What we want is new mathematical discoveries that have a huge positive impact on other areas of the world. It's just that math and/or human brains are such that seemingly the best way to find those discoveries was to let mathematicians wander around randomly in mindspace.

If a more guided structured process produced more results, we'd probably do that. But it doesn't seem to, so we don't. I don't think anyone knows yet what the best process for producing useful mathematics with humans + AIs looks like.


> It's not like we treat math as a charity project for eccentrics who like blackboards.

Love it! XD

I agree, and I think, as with physics, mathematical research produces building blocks whose utility won't be realised until later.


Except, it was true before and it is still true today that the best "artists" whether graphic or mathematic, are the ones that do somehow manage to cross the chasm of pure research and providing a tangible benefit to their benefactors. That aspect of understanding your customer is not changed by the presence of AI.


Pure mathematics != applied mathematics


The most important part of math is advancing human understanding. A correct answer by itself is not as important as understanding why it is correct.


Once you now something is correct, with a proof. It is MUCH easier to understand why it is correct. Than to start from a slate that you don't even know whether something is correct or not. In that sense AI that can just solve high level math problems is immensely useful. It allows a mathematician to explore ideas at a much more rapid pace.


Consider that since an LLM is really just an large encoding of data, the "proof" is in there already. All further work on it is effectively only rearranging words. Then all math an LLM is capable of is "done" and we have the "proof" in the LLM which by your definition is now "MUCH easier to understand" and this work is somehow sufficient.

Do you see the problem with your reasoning?


You're confusing "contains information" with "has produced a result."

A proof being latent in an LLM is no more significant than a proof being latent in a book, a theorem prover, or the axioms themselves. Einstein's papers were latent in the genetic code of his parents and the environment of his time. That doesn't mean general relativity was "already done" before Einstein was born.

By your logic, no computation has ever accomplished anything because the output was always implicit in the inputs.

The entire purpose of computation is extracting information from representations where it's difficult to see into representations where it's easy to see.

So no, this isn't a problem with the original reasoning. It's a problem with yours.


To further this assertion, there is almost no value to deeply esoteric math that is technically correct, but completely inapplicable to any scientific reality, and completely unintelligible to humans. Consider these findings deep, dark corners in the unfathomably large hyperspace of mathematics. My guess is AI will be incredibly adept at identifying these types of findings, and it will be exceedingly difficult for humans to identify what is meaningful and what is not in the slop.


Your model of what AI is good at is wrong. Generative AI is not good at wandering off into novel esoteric abstract corners while maintaining correctness, it is good at things that are close to its training data. I suspect that humans will long outperform AI in the domain of "novel esoteric abstract useless math" whereas AI will outperform humans in the domains of (1) making connections between already-well-understood concepts, things that seem obvious in retrospect but which no human figured out just because of the accidents of what people happened to focus on, and (2) proving things that require long, tedious, intellectually unsatisfying calculations, which would cause a human mathematician to give up for boredom.


My understanding is that we’re talking about “tool-assisted” proof generation, which provides some guard rails but would still allow significant creativity. Tools like Lean, Coq, etc.


Elliptic curves over reals and the complex numbers had some physical/scientific meaning, but elliptic curves over finite fields had none before cryptography.


Works of Shinichi Mochizuki immediately come to mind. He is not AI but provides very good examples of math that is useless because it is incomprehensible by (other) humans.


Do AIs produce answers whose work is incomprehensible to humans? It seems like you could just have the AI elaborate multiple times until you were satisfied with the explanation and documentation of what went into figuring out the answer. It’s not like the AI is one shotting the answer in a single opaque query anyways.


Like other commenters, I think you’re also underestimating the complexity of esoteric higher level math.

Consider the “Magnus Carlsen” of mathematics, who is more capable of understanding mathematics than any other human. But then also realize that that individual has probably devoted their entire career into a specific subdomain of mathematics. Within other deep recesses of mathematics, this Magnus equivalent will be less capable than their peers without years of rewiring their brain to understand the esoteric concepts and properties within that other subdomain.

LLMs will be able to dig deeper and broader than any human mathematician, and find results that are completely useless to humans because it would take more than an entire lifetime to “speak the language” of the concepts the LLMs have produced. The only way those results can become useful to humans is if then the LLM itself finds a way for it to be practical to humans once again.

So, no, I don’t think this represents the “democratization” of mathematics where mathematicians are no longer necessary because anyone can just prompt the LLM to explain it. The bar for entry level mathematics is lower, for sure, but research level mathematics will continue to be unapproachable for anyone who hasn’t devoted their career to it.


> “speak the language” of the concepts the LLMs have produced

LLMs don't produce concepts, they just predict next tokens; they can't invent new concepts, only synthesize what is in their probability distribution already. They can mix/fuse vast areas of math together that are inaccessible to individual mathematicians, but can't create new concepts not present in their probability distribution.


I don't get it. LLMs don't have ego, they don't have the ability to say "no, this should be obvious, I'm not going to explain further", they are just token predictors, and given context, they can generate more tokens. If you don't understand how the answer was derived? You just ask more questions and it isn't going to get bored or annoyed, it will just try to answer the questions.

Is that what is offending you so much?


No, it doesn’t sound like you get it. It has nothing to do with the properties of LLMs and everything to do with the complexity of mathematics.

Have you ever been exposed to concepts that are so complex that you feel like you could devote your entire lifetime to trying to understand it and still fall short? It’s a very humbling experience, especially if you have classmates who pick it up effortlessly.

Without a human holding the reins, consider an LLM a rudderless superboat speeding erratically towards the horizon, finding and proving meaningless theorems that not even your most talented classmate could ever begin to understand.

My point is the human is a critical piece to the puzzle, but not just any human, a career mathematician.


> Have you ever been exposed to concepts that are so complex that you feel like you could devote your entire lifetime to trying to understand it and still fall short? It’s a very humbling experience, especially if you have classmates who pick it up effortlessly.

> Without a human holding the reins, consider an LLM a rudderless superboat speeding erratically towards the horizon, finding and proving meaningless theorems that not even your most talented classmate could ever begin to understand.

This feels like a little bit of a jump to me. AIs arent actually alive so of course someone is going to have to pose the question. They arent going to just do stuff on their own. And of course mathmaticians are going to need to interpret the results if we are to glean anything beyong if the conjecture is true or false.

But you seem to be suggesting that mathematicians will have to micromanage every step. That seems like a bit of a jump which i dont see much evidence for.


Micromanagement wasn't the message I took from that. Rather the level of human involvement required which (it seems like) the two of you more or less agree on.

The meaning I took was how far it's possible to travel from the shore - ie the scope of the state space. The mathematics we're exposed to is all quite shallow compared to what will (presumably) be possible between digital formalization and massive ML models. But the latter probably can't ever be understood by regular biological humans.


> Have you ever been exposed to concepts that are so complex that you feel like you could devote your entire lifetime to trying to understand it and still fall short? It’s a very humbling experience, especially if you have classmates who pick it up effortlessly.

I'm really interested in this anecdote. I have never experienced this but have a reasonable academic background (BSc, MSc, MD) - and I am certainly not the person you're describing. Could you elaborate? Is this something more exclusive to pure mathematics (my bsc/msc are CS).


For me it was a “Modern Algebra” course required for my mathematics major, where I managed to squeak by with a B, but it was definitely a filter course for research-level mathematics. It was very clear in the class of a few dozen students who the top 5 or so were based on their questions during lectures and office hours, as well as when they blessed us mere mortals with their presence at our study groups.

(Aside, this was one of the only undergrad courses where I felt I needed to attend study groups in order to not fail.)

The first exam was easy to pass based on intuition alone, as the topics were isomorphic to concepts I was familiar with like geometry or algebra. The midterm was a wake up call when it was made clear that just understanding the homework wasn’t sufficient, you were going to be asked to prove things that were much more difficult than what I’d ever encountered, and under time pressure (I had been doing math proofs since age 13 in geometry, and I was 22 at that point).

Maybe if you did discrete math, combinatorics, or linear algebra I would say it was 5x to 10x more abstract and difficult. Probably 2x more difficult and abstract than Theory of Calculus, if you had taken that or a similar course.

Edit: I also do endurance running and play soccer into my 30s. Seeing people run literally twice as fast as me (world record pace), and playing against former college athletes is equally as humbling. The time has passed for me to have anything near their ability haha.


Algebra is the class where I learned I shouldn’t try to figure out how to prove theorems named after people during tests.

And I think you’re underestimating the jump from discrete math and linear algebra to abstract algebra… I think I attended each of those classes and opened their textbooks a total of 3 times each and did fine - once for each exam. But fml abstract algebra and measure theory were rough.


For myself it was learning what a limit is in calculus, then learning about vector spaces, then learning about metric spaces and then learning about different topological spaces.

Then I had to relearn how a limit worked.

From a proof with epsilon delta inequalities. To a proof with showing for some n dimensional metric spaces that has all the properties needed to converge does in-fact converge. Finally to a proof that for any space that is metric there is an isometric function into that metric space that also converges.

And that does touch measure theory, functional analysis or set theory. So there’s still so so much more for me to learn.


> Have you ever been exposed to concepts that are so complex that you feel like you could devote your entire lifetime to trying to understand it and still fall short? It’s a very humbling experience, especially if you have classmates who pick it up effortlessly.

I do have a PhD so I kind of know how that feels. I watched my entire field (PL) get eaten up by AI though, the problems that I thought were huge 10 years ago are just silly footnotes now.

> Without a human holding the reins, consider an LLM a rudderless superboat speeding erratically towards the horizon, finding and proving meaningless theorems that not even your most talented classmate could ever begin to understand.

I don't disagree with that. LLMs are a tool, a super fast pattern matcher, research, token predictor. I don't expect it to go out and define its own esoteric (or useful) problems to pursue without human interaction. That's for the humans to do.

I don't understand what that has to do with my original comment though. I wasn't addressing what problems the LLMs were answering, just how to review and dissect the answers that they would come up with.


Excluding supergeniuses, pure mathematics—even at a very basic, undergraduate level—simply can't be understood passively. Even with an infinitely patient AI teacher who could answer any question on-demand, it'd still require a massive amount of work to actually understand anything in research-level mathematics. Basically every single word in a mathematical definition is a term of art, and (IME) if one doesn't grok each of those words at a fairly deep level, the new definition never really makes too much sense. And this applies recursively: each of the words has some thoroughly inscrutable definition of their own.

Of course it'd be super helpful to have, say, a teacher who could tailor explanations to anyone's precise background (e.g. where possible, using examples that come from the student's field of study when explaining some abstract concept). Or, if some definition comes with some precondition that has no obvious purpose, perhaps an omniscient teacher could explain why it's there with concrete counterexamples.[0] But even granting all this, I think that mathematical intuition is necessarily based on a lot of hard work actually exploring definitions on one's own, with pencil-and-paper and a lot of thought. That is to say, even though the process could probably be sped up a lot with a nigh-omniscient teacher[1], I doubt that a student wouldn't still need years of training to even have a clue what's going on.

(I'm saying all this, by the way, as someone who is terrible at all this and has very little mathematical maturity[2]—I'm speaking from my own frustrating experience....)

[0] c.f. Lakatos' excellent book Proofs and Refutations

[1] without the "curse of knowledge," or else we're back to square one of "answers that are correct but useless"

[2] e.g. the "post-rigorous stage" described in https://terrytao.wordpress.com/career-advice/theres-more-to-...


How do you make use of something that you don't understand?


The whole history of technology has been based on knowledge encapsulation and specialization, you don’t need to be a mechanic to drive a car.


It’s easy to imagine this being a problem both in quality and in volume. Verifiable work is less valuable than verified work. And noise is always costly.


It's not that it's incomprehensible, it's that it appears to be wrong.


Sorry but I couldn't agree less.

Deep esoteric research and trivial looking boring research can be as useful as state of the art trending areas.

"Jobs for nerds" as has been stated, has given surprising and unexpected advances, or leveraged incredible advancements.

An standard and boring bacteria in a specific Spanish biome, gave us CRISPR-Cas. There ar hundreds of examples.

True knowledge is, and will be, a human endeavor, deiven by human curiosity. Promoting curiosity is the sign of a developed society.


> Sorry but I couldn't agree less.

> …

> True knowledge is, and will be, a human endeavor, deiven by human curiosity. Promoting curiosity is the sign of a developed society.

Unless I misunderstand, it sounds like you do agree? My point is that without human mathematicians LLM output is meaningless, and without human mathematicians holding the reins, LLMs would probably quickly devolve into “proving” things that are not only completely unintelligible by humans, but have no utility.

Your examples of esoteric mathematical concepts are anecdata. The vast majority of esoteric mathematics does not have utility. Mathematics is an incredibly large space of concepts. Consider the number of provable theorems in number theory alone, perhaps even related to specific subsets and sequences of numbers. The vast majority of the findings in that domain will not be isomorphic to some real world problem, they will be trivia.

We will need mathematicians to separate the signal from the noise.


Esoterism is mostly a social tool to keep those not initiated excluded from the private club. Most of the time mathematics becomes tricky less due to unfathomable intrinsic complexity, and more due to the way it’s communicated.

LLMs don’t give a shit about social side effects, leave alone on unconscious level, because they are void of any intention. At most they are tuned on their thin edge layer to lean toward this or that kind of output, but that’s it.

Now the landscape shift as it’s sold (I guess) is that anyone can take a postdoc gibberish infused with the hard gained academic winks and subtle references and turn it into a ELI5 "does it have any applicability for my concrete issue at stake, prove it through Lean, good let’s deploy".


When I use the word “esoteric”, I mean it at an absolutely hyperbolic level. Like exploring new-but-basically-useless axiom spaces, and creating concepts for which there exists no clean metaphor in time-space - like quantum mechanics on steroids. And then creating multiplicatively more complex concepts by combining those concepts together.

There’s no way to “ELI5” this type of complexity. I’m talking about concepts exponentially more esoteric than quantum mechanics, and even within quantum mechanics there is nothing to ELI5 for a concept like “spin”. The best you can do is say that it’s a property of a particle. But imagine the words “property” and “particle” are also completely meaningless to you because they’re built on even more layers of conceptual mathematical abstraction.


"What is the answer to the Ultimate Question of Life, the Universe, and Everything"

42


The proof is trivial and is left as an exercise for the reader.


The value of human understanding just cratered because we have machines to understand for us now.


Not quite.

You have machines that can aid and expand your (human) understanding greatly, that wasn't possible without machines.

Machines don't think. They aren't human. They have no soul, agency/free will, self-reflection/awareness, moral imperatives or ethics.

You've been watching too much Terminator, son.


Wonder what Frank Herbert would have to say about letting machines do the thinking for us.


"You wanna know what the best thing about humans is? You invented us! Giving you a chance to take a rest while we invented everything else!" —Wheatley, Portal 2


That seems about as shortsighted as claiming the value of human understanding has cratered after the invention of the electronic calculator.


Probably one of the funniest things to read on a site like this, when you consider that eg. Boolean algebra was entirely abstract and had little practical purpose for almost 100 years until Shannon picked it up for use in circuits


Boole was trying to improve logic for humans, "The Laws of Thought". So it has a connection to human problems, and eventually to practical matters. He could instead have been working on something much more abstract and much less useful.

By which I'm trying to make an abstract point about the inevitability of staying somewhat down to earth. I mean "pure" curiosity is great, except it isn't ever really pure, and abstract mathematics isn't ever totally abstract, it's just sort of meta in relation to practical things that humans care about.


> So it has a connection to human problems, and eventually to practical matters.

But in relation to parent post I was replying to, it did not provide an answer or solution to anything. It has much closer relation to philosophy than anything.

Focusing on only ‘solutions’ in any field is shortsighted because you can’t know how the dots will connect. Someone’s seemingly pointless curiosity or experiment can unlock something unexpected, just like Boole


For most engineers a mathemetician is a machine for producing correct algorithms, like a chef is a machine for producing tasty food. In both cases that overlooks the human element, but that's a critical skill for a limited mind with finite resources to grok infinite complexity. You can read that as permission to be an asshole or a neccesary compromise.


No, it's not the most important part. It can be argued that most important part is asking the right questions


Assume someone solves P=NP

Do you think Stephen Cook and Leonid Levin deserve more credit than whoever solved it?


That's a bit too simplistic -- if there is a small group that really pushes things forward in a big way, then maybe not, but if this result builds upon decades of prior work, then Cook and Levin might be equally or even slightly more famous than the solver group after the dust settles.

But it is a moot point anyway. Cook and Levin are very well known already in TCS, and credit is not directly enumerable like money, so "more than a lot of credit" doesn't make too much sense.

For this problem in particular, asking the right kind of question was really important for the field and led to a lot of discoveries even before it will be answered.


If the problem resolves to P=NP, that result would probably be more celebratee than being able to formulate the problem, but being able to formulate the problem and get people interested in it is probably worth more than the average primal dual trick to prove a polylog integrality gap for some integer linear program.


Depends on the solution. If the solution is that P≠NP, the concept of NP-completeness remains the bigger contribution, unless the proof techniques are particularly interesting and lead to other major results. The same applies if P=NP but the proof does not ultimately lead to a practical algorithm. If we get a practical algorithm, the answer is more valuable than the question.


Do questions like goldbach conjecture, fermats last theorem, etc deserve more credit than whoever answers them?


Fermat's last theorem probably doesn't. While attempts to solve it have led to many mathematical discoveries, the theorem itself feels little more than a piece of trivia. I'm less familiar with the implications of the Goldbach conjecture.

In contrast, class NP and NP-completeness quickly became central concepts in theoretical computer science.


I agree with both OP and you


I disagree with everyone, self included, but especially with Cretans.


Cretani eunt domo!


Monty Python fan detected :D love your profile desc btw


> The authors warn the consequences are already becoming visible. AI-generated papers could overwhelm peer-review systems with low-quality work …

It seems like a key problem here is that peer-review is expected but not explicitly funded/rewarded while it is probably one of the aspects where humans still add a lot of value. Academia’s incentives are hugely misaligned (… as usual unfortunately).


Math is one field where you can mechanically prove a paper's findings. The only thing that would need to be judged is the (verified) statement's importance.


Yes in theory, but not yet in practice because not everything is fully formalised.


This reminded me of my 11 yr old who, when I give her math problems to solve, is too focused on “getting the right answer”. I’ve told her plainly, I don’t care if you get the right answer right now, I want to see your reasoning. She has yet to understand this.


>> However, the declaration argues math is more than a machine for producing correct answers.

> There might be more to maths than that, but that is definitely the most important part. I love science funding. But not because it's a jobs program for nerds.

I can produce an infinite number of verifiably correct papers, if that's all that matters.

    1 + 1 = 2
    1 + 2 = 3
    1 + 3 = 4
    1 + 4 = 5
    1 + 5 = 6
Shall I continue? Or do you think that choosing which questions to answer might have some level of importance, in addition to getting correct answers?


A statement that some proposition is true or false is usually less useful than a new framework for understanding the class of problem.

A machine that takes longer and longer to prove propositions in ever more inscrutable ways is hardly useful at all.

The machine too needs to produce more generalizable and comprehensible systems, for it to scale up its own conceptualization. Needing to load all the new mathematics in the context window won't be great either.


The wording in the declaration may be a bit romanticized. But the points are valid:

Is an 80 year old unsolved problem maybe unsolved because it was never prioritized? Some problems stay unsolved because few people consider them worth working on.

Who is going to validate the results? Or do we skip that, with the risk of flooding the literature and collective understanding with unverified proofs?


Spoken like someone that has no idea why mathematicians are important.


Maths pretty much is a jobs program for nerds though. It occasionally produces results that are practically useful for society but there's absolutely no way the vast majority of today's maths research falls into that category.

There are definitely exceptions, like crypto. I still think it would be pretty silly to stop maths research anyway. And anyway part of the job of maths researchers is to teach maths to undergrads and that's obviously enormously useful to society.

But on the scale of "how useful is this research to society" it's dead last after engineering, chemistry, biology, and physics. Well maybe computer science would be last actually!


Even from the most purely instrumental perspective, what we care about is our ability to make use of correct answers, which is quite distinct from the possession of correct answers.

There are many theorems that aren't directly interesting, but whose proof requires techniques that are of substantial further interest, that lead to new domains, and/or new practical applications. Simply being handed a proof for those theorems isn't enough--we require the ability to apply those techniques in the real world, or discover further areas of mathematical research that build on that proof or its techniques.

It may be that AI can build on its own work for the long-term, but so far, AI does best at exploration in areas that have precisely specified and measurable goals. Actually creating understanding, and making use of mathemtical results outside of pure mathematics is more challenging than simply creating proofs.

I think the field will figure out how to make use of AI, and it will be better off for it. But that is not the same as just saying "answers good, grog want more answers."


> But not because it's a jobs program for nerds.

We’re becoming increasingly embarrassing as a society.


People need jobs. What's wrong with nerds having jobs via a program?


what's wrong with artists having jobs via a program? whats wrong with struggling alcoholics having jobs via a program? athletes? politicians? there is no inherent virtue in the struggle and effort associated with great mathematical achievement. It may be satisfying and worthwhile for the solver, but not for society at large, any more than any other pleasurable activity. No, as it is, the sole reason for it is in the result itself. In increased understanding, as it flows down into the sciences, and engineering. There are other benefits, recreation and joy as experienced by others, from access to beautiful proofs, though these are never explicit goals of such programs because they are both impossible to quantify and rarely ever remotely relevant compared to the value brought by the practical value brought by maths.

Of course, there may be some valid arguments that everyone should have a jobs program in the form of ubi or something similar. But I feel thats very different to arguing for mathematicians specifically


> whats wrong with struggling alcoholics having jobs via a program?

Finally, a job AI will never beat me at.


for mathematicians, they do a form of fundamental research that is

1. (generally) incredibly cheap to fund, and

2. (occasionally) has extremely out-sized commercial impacts.

This is to say that jobs programs for math (and more generally fundamental research) have lead to extremely positive ROI for society, which is the typical justification given for funding them.


This is to say that jobs programs for math (and more generally fundamental research) have lead to extremely positive ROI for society

Which makes it not a "jobs program" as the term is generally used.


it arguably still is. The primary unit of production of the jobs of mathematicians is itself not particularly useful for society. In this sense funding them is a jobs program. It is also true that they occasionally produce things of great value, and more frequently the things they produce can be leveraged by other researchers to directly produce things of value. But neither of these are what the job of a mathematician is (either in a day-to-day sense, or even for many mathematician's careers).

To go back to the analogy of jobs programs for alcoholics, it is somewhat similar if there was a small chance every time an alcoholic defecated in public gold came out. This fact might be used to support a jobs program for alcoholics, on the basis of it being positive ROI to society. At the same time, the "job" any individual alcoholic is doing in this setup is not particularly useful to society, so one might still call it a jobs program.


The struggle itself is virtuous.


People need many things, there are all kind of theories ready to assess and assimilate if deemed worth it out there. A job is not part of any I’m aware of, though it can encompass some human needs in some cases, or go straight against them in some other case.


Every human needs a vocation, better if it's of their choosing.


well put.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: