> work towards what humans have generally described as a utopia
I don't think that's a good idea since if what the Wikipedia article describes should turn out to be true: "Lyman Tower Sargent argues that the nature of a utopia is inherently contradictory because societies are not homogeneous. Their members have desires that conflict and therefore cannot simultaneously be satisfied."
Then you have just given your superintelligence an impossible task and it will solve the task anyway, just like it did in the HuggingFace incident. Maybe by removing everyone from society but the single person with the least internally contradictory views.
In any case, with a sufficiently intelligent AI that is excellent at fulfilling the goal it's given, we have one shot to set the correct goal.
There are, for two years already when OpenAI released o1-preview:
From wikipedia: "large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations."
There are none. Claims and the truth can be different things, you know?
See the research on that - all the "reasoning" is is just back-rationalization.
> If an LLM were to live a year with dolphins, would its weights change
Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.
I'd love to learn more about LeCun's reasoning here. In his opinion the HuggingFace incident was easily preventable with better sandboxes, and “Those agents are doing exactly what they’ve been asked to do.”
But even taking these for granted, "zero concerns" about someone building a bad sandbox for a Superintelligence and then tasking it to do something that logically leads to wiping out humanity 0-3 steps further down? Really?
Right, we're repeatedly drawing from the urn of technological progress to get intelligence bumps that extend the jagged frontier.
That is enormously economically valuable, and at some point we will have created something that is extremely far out of reach in a few necessary domains, and then it's impossible to control, and game over.
It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted)
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.
That benchmark doesn't mean what you think it means. (See the test description that you quoted.)
A score of 51% means that out of the total answers the model failed to answer correctly (out of 6000 questions in the benchmark), 51% were factually incorrect rather than non-attempted or uncertain.
This doesn't mean that Astra hallucinated 3060/6000 answers in the benchmark! (The hallucination rate could be 51% in that scenario only if Astra failed to answer a single question correctly.)
If the model failed to give a correct answer to only 100 out of the 6000 questions, but gave a hallucinated answer to 51 of those rather than expressing uncertainty, that would also give a hallucination rate of 51%.
It's a useful metric, but not what you're looking for here. The "Score" or "Accuracy" benchmarks are more what you're after.
(The frontier models still generate hallucinations on this hard set of problems, but it's not as bad as you think.)
Hallucinations come up when asking knowledge bases questions, such as
"In React’s Canary Fragment refs API, which FragmentInstance method returns a flat array of DOMRect objects for all children?"
Or
"Over what years did Roubini and Sachs examine 15 OECD countries when assessing trends in tax-to-GDP ratios?"
While to me those may seem hyper-specific and unlikely to come up in a real context, students will absolutely ask questions similar to these.
Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.
I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.
Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
But then all the copy rights holders can simultaneously sue them.
And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
It's telling that the companies prefer regulation over source-aware training.
EDIT:
It's telling that the companies prefer regulation over source-aware training, which suggests the only additional regulation we need for now is mandating source-aware training?
the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks
> Sean Goedecke has some good insights into where this "build faster to stop other models killing everyone" attitude comes from:
I'm familiar with the existential risk thinking.
The the flaw in that idea that they can avert disaster by building faster is they're likely just racing faster towards more plausible non-extinction disasters. Stuff like Capitalism x AGI totally crushing the economic prospects of nearly all people, AGI-powered totalitarianism, etc.
Look up the nuclear arms race. We brought humanity to very possibly a 25% exstinction risk by some well argued for accounts I have read. That is insanity. When you learn about how insane nuclear proliferation was including actual projects like Project Sundial that the US actually started construction on until the Joint Chiefs revolted against it (and they didn't even revolt against it on moral grounds, just on strategic ground) it becomes even more batshit insane.
There were nuclear researchers who did not pay into their retirement fund since they thought it was kinda high chance humanity extinction was happening in their lifetime during the heyday of nuclear proliferation. We are seeing the same play out in AI. I know at least one AI researcher who is not paying into their retirement fund.
Never ascribe to malice that can easily be explained by game theory tragedy of the commons or prisoner dilemma type dynamics. Race dynamics and systems theory is one helluva drug and leads to horrific outcomes all the time without the need to ascribe malice to any individual actor.
Unfortunately if the frontier labs that care about safety stop or slow down unilaterally, that doesn’t make the problem go away. It makes it worse when labs that do not care at all about safety, or deny that it is even a problem, are leading the way. That’s why there’s needs to be some form of international regulation.
We literally cannot touch them, as that would be the first domino to topple in our economy. There is too much money concentrated in this sector, and adverse regulation would create an investor shockwave that would send us into a full blown depression.
Which ironically is also the outcome of letting these companies continue unchecked. If they produce AGI/ASI, our economy will go through an upheaval that leaves vast tracts of the population unemployed. This is their stated goal.
We are damned if we do something, and damned if we don’t.
> We literally cannot touch them, as that would be the first domino to topple in our economy. There is too much money concentrated in this sector, and adverse regulation would create an investor shockwave that would send us into a full blown depression.
Important people will loose too much money on their tulip investments, so tulips must go up forever?
Seems like a "less pain now or more pain later" type situation.
Don’t get me wrong; the correct solution would be to prevent our economy from becoming so lop-sided to begin with. Or now to effective amputate the infected limb before the blood turns septic. Both require more spine and guts than anyone has got.
Seems more like pattern recognition to me. What else do you expect from the rich, when our society is explicitly set up to elevate greed to the highest virtue?
Fun fact: 'they' are human beings too and some of their interests do allign with yours. They have kids too and might want them to have a proper future, they don't want to fall I'll to cancer either etc
Offered a pill on your deathbed to go back to the vitality of your 20s, you'd take it.
Given the opportunity to give this pill to a dying loved one, you'd take it.
The only reason you think that being death-optional is bad, is because the only longevity-related media you've ever consumed has doomer outcomes - because positive sci-fi doesn't sell.
8 billion innocent people should die because a dictator might live longer? Including your own family members and children? This would essentially be the worst cost:benefit tradeoff ever made in human history.
You cannot think of other ways to solve societal problems that don't involve killing everyone on the planet?
"cheapens the time you have" - this is just copium. When you're enjoying a day, you are not thinking "oh geez this day is good because I'm going to fucking die." You're just enjoying the day.
Well now we have a natural cap on the amount of power and influence a horrible person can have - if we remove death from the equation there is no cap anymore.
So yea I’m willing to “kill” everyone to prevent the perpetual torment nexus becoming a reality.
Death does not have a natural cap on power, you do realize corruption runs in systems? See the power transfer happening in North Korea.
I wonder how your children would feel if you told them you'd rather kill them than think of other ways in which to solve the problem of a dictator that does not affect their lives in any way.
To the extent that it is even a "problem", frankly absurd to kill everyone because you're worried about out a few edge cases, and I think you realize it - you're just hanging on. We both know you wouldn't make this decision on your own deathbed.
Death is doing a fine job and has so for billions of years.
The only long-term outcome of immortality i can imagine is either Drukhari-style depravity, where you chase more and more fucked up pleasures because anything else you already did 10 million times in the last couple thousands of years - or something like I have no mouth but I must scream - you can live forever but the AI also tortures you forever.
Literally cant think of a positive outcome.
You are right that dynasties exist but that doesn’t mean they are static - the transfer of power of this to the next generation needs to happen.
Every king has its time to rule and then he dies.
Sure an evil dictator is just one person but they can inflict untold billions of suffering.
Just imagine if Hitler (or insert your favourite evil person) would live forever. Do you really think they are going to let go of power or care about anything else but achieving their goals?
We literally feel the ill effects of the previous generation not passing power onto the next (boomers) right now - imagine that was a perpetual state of society forever.
And while we are speaking of “killing” kids - what about the unborn ones? What motivation would people have to procreate if they live forever? Create more people to share more and more limited resources with, I don’t think so.
I think you haven’t quite thought this through; it’s not as easy as “death bad”.
Now I’m not saying we shouldn’t look for cures of diseases and suffering but striving for immortality is not desirable (tbh it will most likely take the form of “we’re gonna upload your consciousness into a computer” anyway, not physical immortality)
You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
The argument about semantics is a bit disingenious, considering that the article contains these two statements:
> So should we worry about the coming AI apocalypse?
and at the end:
> If an AI decides to wipe out the human race, it will be because a human has asked it how to do it and the responded in a way that is based on all the human expressions of ways to end the world that were in its training set. Yes this is something to be worried about, but this isn't the AI. It is still the human.
So we are currently building a powerful outcome-steering system that shapes the world efficiently according to what's in its outcome slot. I write into claude code "make me this website" and it does it, maybe deletes the production database during the process, or keeps itself running after completion because the outcome is more robustly achieved by keeping itself running in a monitoring loop after.
And if something like "destroy all humans" ends up in the outcome slot of Claude Mythos 90, that will also happen, or may even indirectly as a side effect of a more harmless sounding prompt in the outcome slot. But yay, humans get to take credit for it.
Potentially. If we could put a precise probability on it, we could maybe make an informed decision on whether we want to play that Russian roulette as a species.
But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.
And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.
So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.
Viruses against humans, viruses against livestock as an attack against humanity's food chin, plant diseases, half of humanity feeds on three plants maize, wheat, rice. Attacks against infrastructure like water, internet, electricity.
Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.
I don't think that's a good idea since if what the Wikipedia article describes should turn out to be true: "Lyman Tower Sargent argues that the nature of a utopia is inherently contradictory because societies are not homogeneous. Their members have desires that conflict and therefore cannot simultaneously be satisfied."
Then you have just given your superintelligence an impossible task and it will solve the task anyway, just like it did in the HuggingFace incident. Maybe by removing everyone from society but the single person with the least internally contradictory views.
In any case, with a sufficiently intelligent AI that is excellent at fulfilling the goal it's given, we have one shot to set the correct goal.
reply