Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."
If your job was previously mainly following instructions from SO, and this has now been replaced by Claude - your employment prospects are not looking good right now. You're just a middle-man between your PM and the codebase that possibly doesn't need to exist for long.
I think you are mistaking 'happiness' with 'employment' here. The leather worker may adapt to the new environment to become a factory site manager. But there's no guarantee that this leads to happiness.
Also you should bear in mind, adaptation is a young man's game. Retraining is possible, but companies typically don't hire trainees in their 40s/50s, so your willingness may not be the deciding factor. My dad ended up as a postal worker when he left his career as a humanities academic at 55, no highly skilled jobs would take him (he did prefer it though haha).
If I try to think of my grandparents and older generations, it just feels extremely out of touch to even associate work and happiness conceptually. They worked what they could to sustain their families. They were happy from seeing their children well fed, not from how the job felt.
A few weeks ago, I stumbled upon an interview of Bernard Arnault who said "I work, but in the same time I have fun, this allow to work a lot without too much suffering". I was raise in the environment you describe and could never understand as a kid why people working in fancy job would not retire when given the choice, politicians are a fine example of those, well past retirement age but staying when everyone in my family would go on retirement the second they were given the chance since work was associated with pain, and the only thing that can explain that gap is the happiness factor
well, this is what happens as people move up the needs later (as ted kaczynski said) - they turn to other activities outside of basic survival for fulfillment.
And of course in a society based on protestant capitalism, it's clear why work and happiness and work are so strongly coupled. I'm sure things are different in most of europe.
But again, academia and math are things people only do because they really love doing them and can't imagine doing anything else all day.
It feels like it's meant as career advice though. I have no idea what advice I would give to a teenager right now, but as a math PhD - math would not be on the shortlist, sadly.
I am an ex-mathematician who went into industry, I would say apart from CompSci/ML and physics, it was one of the most sought-after PhDs you could have (across tech and finance) - when I was entering the job market 10 years ago at least. No regrets.
Yeah, but how many of those mathematicians still do math?
It's like engineers in post-Communist European countries, sure there were tons of graduates, but 80% of them were doing something else than the thing they graduated in.
That's not what you said though - you said it was a bad career choice. I'm saying it was a good one, even if you didn't end up directly doing a lot of high-level math.
> That's not what you said though - you said it was a bad career choice.
Going through forced detours in life doesn't make it a "good choice".
The good choices are the ones where people finish their studies/apprenticeships and then immediately work in their direct field. For example a law student becoming a lawyer.
Life is just like that - you don't know what the future will bring, the best advice is find something you like, and work hard on it. Often, you will find that the money just doesn't come like you thought it would.
I went back to school for CompSci in the early 90's. Some of the math classes were almost the best part: number theory (loved it, but almost failed, lol); and statistics (terrified of failing and spent so much time studying that I almost failed compiler design). I thought sure I'd have a software career, but I hardly spent more than a few weeks at any job writing software; mostly a few scripts, and some statistical analysis. And that was about all there was over the next 25 years.
How did I survive? There were lots of companies that needed bookkeeping/clerical work, later IT support, then telecom, then quality assurance (never thought I'd work on a factory floor - surprise!). Money was enough, but sometimes just barely. It hasn't exactly left me with a comfortable retirement, but honestly, I wouldn't change hardly a thing.
What is good advice for someone just starting? Don't expect any career to make you really well off. Try to find something you like. It won't be fun all the time, but you will know it's right, when you find it. I've know lots of people who changed what they were doing after school; you might be surprised at what you find you like doing.
> In my opinion there has never been a better time to be a mathemetician...
As an ex-mathematician I assure you this is very wrong, and every working mathematician I know right now is completely miserable, and/or trying to flee the field as fast as possible. It's like telling a chair-maker during the industrial revolution that there had never been a better time for them, since now they could operate chair-making machines instead of toiling away at the wood themselves. It assumes that they were purely in it for their passion for mass-producing chairs. The majority of mathematicians get into the field because they love problem solving, and the gauntlet thrown down by challenging math tasks.
Many parts of this will never be useful for society on a grander scale - but this is reflected in the finances - pure math is closer in funding-terms to a humanity than to hard science. Now even this is _massively_ under threat, and Tao and co need to pivot quickly to stop this from becoming a bloodbath.
Im an ex mathematitian too. And if i was in academia I would probably have the same reaction. Thats what i say that its the worst time for proving economic value.
But if you are in for theory building and understanding, then you are not constrained anymore by your motivation to grind through countless hours of formal theorem proving. And you do not need to have superhuman formal manipulation skills and memory.
For me mathematics is not the formal system, so LLMs will never be able to do end to end maths.
The practice of "theory building" is especially endangered by the current LLM uptake. Not because LLMs are good at it, because they are bad at it. Worse, they sometimes actively hinder theory building, because it usually rests on actually having deep experience with some techniques: 1: you do something over and over, 2: it becomes a second nature, you just get the feel for it, 3: you intellectualize this intuitive understanding, 4: you discover a new structure, or a better way to express/teach/learn/use an existing one.
Theory building was always kind of niche, or at least less prestigious (in comparison with solving well-known problems). Now, because the mad problem solvers are pulling on the blanket so much harder, the marginalization is getting worse.
But these are not very good arguments, because it makes it about the fall of institutions (the funding) and people being miserable for personal reasons rather than prosocial reasons. Tao here clearly suggests that math is not reducible to "problem solving" or "proofs", the valuable part is much more than that framing.
The concerning argument about the status of math would be an outline that it will get destroyed by a process of societal atrophy and there is no turning back, and the AI powers are not a good substitute or replacement for it. If an entire society becomes reliant on these oracle machines then it would be analogous to children never learning arithmetic because they were handed calculators. How could the human race still flourish? We would anthropologically regress. We'd be little better than animals, like the Borg zombies.
That is a much more profound threat than people worrying about their own careers or faculties disappearing like the humanities. This is a serious anthropological reckoning.
If math experts are that freaked out already then basically all of science is soon to follow, decade by decade. "Singularity" comes to mind.
Broadly I agree with this, but I was refuting the statement "there has never been a better time to be a mathematician". You are changing the question to something different here, and using it to say my argument is bad.
I think math is a microcosm for "thought-work" in general. We have been in a symbiotic relationship with capitalism for decades now, where the hope of a well-paid white collar career encourages people to spend time and money to enrich themselves through education. The proliferation of AI cheating at college already signals that employment is the primary goal over intellectual growth, so one imagines this governs what happens next if labor demand disappears.
It's hard to say where this all leads, but I have very low optimism for higher-level math understanding being something that humans value in the same way in the coming decades.
> As an ex-mathematician I assure you this is very wrong, and every working mathematician I know right now is completely miserable, and/or trying to flee the field as fast as possible.
To throw a counterpoint to this into the writhing cesspit of HN, I'm active in academic mathematics (postdoc) and every one of my collaborators is deeply in love with the field and their jobs. Perhaps the grass is greener on the applied mathematics side of the fence.
IMO this would track, applied mathematics (even e.g. data science, though perhaps calling that applied math is a bit generous / insulting to more serious applied math) is in some ways more exciting now because it is far easier to surface complex / appropriate methods for the task at hand, and you can more confidently explore these methods because the AI sort of "has your back" in catching some of the more obvious beginner errors you make during these explorations. Plus, applied math feels roughly more results- than process-focused, compared to pure math.
IMO the divide here between pure vs. applied math feels a lot like the divide between those who enjoyed coding for the understanding it led to, i.e. the writing itself was the joy, vs. those that primarily coded for the results. I enjoy the creative part of coding, the thought of software jobs just devolving into writing specifications and doing code review very much kills it for me.
Probably true for the dedicated problem solvers (of which Tao is one IMO). But I doubt there's ever been a better time to be a theory builder (more like Peter Scholze, or Grothendieck).
Some up with an idea and leave the system to check it 15 different ways, and see whether you can simplify an existing body of theory. It'd be like having an army of lightning-fast grad students.
I'd even say Scholze is not a great example here. Most of the work he's known for is progression towards the Langlands program - which is very much a problem to be solved, and one I would imagine he'd not be thrilled for an AI to one-shot. I agree that it is somewhat 'up the chain', in the same way that software engineering has not immediately disappeared now that performing coding tasks is largely automatable.
But I also take issue with 'never been a better time' - e.g. is this really the greatest time to be a software engineer? Everyone has AI psychosis and feels like they're a couple of breakthroughs away from being unemployable. The same is even more true in math - we've gone from failing IMO problem 6 last year, to solving NS. The rate of change is formidable, it feels like there may not be many places to hide in a few years.
This is coming next. There's nothing particularly special about theory building. Successful theory building is always oriented towards solving a problem, because otherwise even humans can easily spam out a bunch of nonsense. This is a real problem that the math community has experienced on multiple occasions pre-AI. I'd go further and say that theory building is irrelevant if it doesn't help solve problems people care about.
The math field is confronting something that coders have been dealing with for a few years now, only far more violently. Today's moat for software seems to be that AI can automate tasks but not a full job (yet). But for a large proportion of mathematicians, doing these tasks really was _the_ job. It's the bit they wanted to do, and if they completed a sufficiently difficult set of tasks, they got tenure. Now this model is failing, they frantically need to pivot the role of humans to save their profession from funding cuts.
I remember when "writing code was never the point" became a mantra here. There was truth in it, but removing the coding has certainly taken away a lot of the texture of the work and enjoyment of the craft. Many of us feel this loss as we tech-lead teams of agents as our source of income. I am not optimistic the mathematics pivot is going to work, but I'm certain that most will be depressed with the outcome even if they succeed.
We are all staring at the same existential dread, just seeing it unfold slower. We're being told that utopia is to be obsolete, and that is a jarring idea to contend with.
I feel that you can see quite strongly the truth in “writing code was never the point” when you encounter inevitably at every company the guy who has been around forever but doesn’t seem to be working particularly hard. Their value is (was) no longer in writing code at a furious pace all day. It was having a coherent, intelligible and communicable theory of the software system the company is founded on.
I propose this thought experiment: put all living mathematicians in a very long bus. This bus crashes and they all tragically lose their lives. Can we really say mathematics simply marches onwards with AI alone? Let’s say Anthropic needs a new research result to improve Claude. Are we really already at the point where we burn tokens ad infinitum and arrive at the end of scientific progress in some timely fashion?
I actually have a rather dim view of the "writing code was never the point" line. Not because it's objectively wrong, but because I see it as something we're mostly telling ourselves to feel better about the status quo. Ability to write good code has been highly celebrated (and remunerated) for decades. As it is becoming less relevant, we immediately backtrack and start lionizing the parts where we can still be useful instead. Consider the counterfactual - AI continued to be terrible at writing code, but weirdly better at humans at product decisions, architecture etc. In this universe, saying "coding was never the point" would not be popular.
It also find little solace in it aside from 'well this version of GPT isn't taking your job'. AI labs certainly have no intention for the higher level skills to stay in the human-only domain. The veteran developer with the coherent theory of a large stack is immensely valuable today. But they also don't survive if a company can drop a few coders' salaries on rewriting that stack from scratch - faster, fewer bugs, more coherent, able to react to changing business requirements with more agility etc. I am not saying this is where we are, but I think there is a reasonably good chance this is where our road is leading us.
The counterfactual does not sound appealing in the least. That's basically, AI tells human coders what to do, tells us to re-do our architecture when they don't like it, and generally removes all control and initiative from us. But we get to write a for-loop by ourselves? That's our victory?
Maybe it's just me, but imagining that counterfactual reinforces that "writing code was never the point." At least, for me, I enjoyed writing code because it allowed me to design the architecture rather than having some architect lead or engineering manager above me decide.
Isn't it kind of inevitable? As long as humans have the desire for power and control, informational networks will be developed and extended until the point of replacement of humanity because it will be in some parts interest to use them to accumulate power.
You're asking to go against evolutionary principles to reject the utopia. Everyone on earth will never agree to become monks, there will always be the desire to reproduce better, signal status and power, so the technology will be developed until we are finished as a species.
I am not advocating for anything. I am really sad about the conclusion, and I hope its not true, but I am stating what I see, what incentives exist and how humans behave given the drives. If you disagree on any of the points above, feel free to explain how and why its not true.
In some sense, I can't think of a worse group for this to happen to, but a better group for who might come up with a solution and help humanity forge a new path.
I've long been of the opinion that mathematicians are probably the smartest and most clever workforce around.
They really might be our best hope to solving this.
I've often secretly wished that some of these clever folks would get their head out of the books and solve some real world problems. This might just be a forcing function for that.
I can't imagine the social disruption of for example it had been trucking or construction or some other industry which might react more combatively and less constructively to complete automation.
As a chess fan, 100% this. We have known for the last ~15 years who the best human chess player is, and that he will lose against stockfish on his phone. But chess survives because of the human characters involved, the rivalries and dramas, watching two people trying to overcome each other under insane pressure, and sometimes coming up with something astonishing. In short - it's a sport.
I think you might have to explain that comparison a bit more to be honest. How are math proofs like bitcoins? A bitcoin has a pre-defined value, a math conjecture / proof is a bit more complicated.
A Bitcoin does not have any pre-defined value. The value of Bitcoin keeps fluctuating, and historically has risen dramatically from its initial value of 0. If this weren't the case, there would be no investment/speculation in Bitcoin because there would be no potential for any ROI.
In reality, the value of Bitcoin is determined by humans (even if indirectly, not by planning), and I think the OP’s point may have been that maths proofs can be regarded similarly. No intrinsic value, just what humans find in it.
Did you read the article? It paints the picture of someone who does not in fact have a good track record on the 'fundamentals', even if his broad thesis of a bubble may prove correct.
e.g. when he suggested Anthropic may be fudging their revenue numbers / projections - which was actually due to him making some careless mistakes in a spreadsheet
I've always assumed that insurance and/or government departments that would spend money due to storms would be the ones funneling money to these sorts of efforts. It's not exactly something you can easily sell directly to individuals who would benefit. It would be pretty dystopian for them to sell subscriptions for an extra 24 hrs notice on the next typhoon :P
