At work we have rather strict rules for how to run AI agents, and for good reasons apparently. The problem is that once these rules are in place and enforced, using an agent is just more hassle that I feel it's worth. The agents work in complete isolation in a VM with access to pretty much nothing. Some colleagues are making good use of them, but just looking at the policies and tooling we have put in place makes me give up in advance.
Same reason why I'm upset when Temu ships a garbage plastic gadget. Someone must have wanted it, they ordered it, but I doubt that they wanted the quality that showed up. We've all seen the videos of what people think they order vs. what actually shows up. I don't get why you'd go through the trouble of wasting the resources on a scam product, rather than just outright robbing people.
What if the Temu plastic gadget that you ordered actually turns out to be actually quite good and does what you’d expect it too? I think that’s the real comparison here instead of automatically assuming her books are crap that no one enjoys. Obviously enough people enjoy her books (some of it anyway) to make her money.
Are there reviews though? Many readers are always ready to publish reviews of stuff they read, so it's mostly just a matter of finding reviews of works by this author.
Imaging if people returned all the defective junk they got from Temu or Amazon. If the price is low enough people aren't returning anything, it's simply not worth the hassle and they'll accept that they got swindled.
For books I'm not sure, assuming that the author actually have a story to tell, but is just lacking the skills to write a book, I don't know that I think you can justify calling it a grift when they're trying to write that book using AI. 370 books however.... Is she really that imaginative?
There's a hell of a difference between having an AI help you write a fictional story that you've planned out and dreamed about for years, but just didn't have the skills to write and then pumping out 20 books a month. There's no way those were have been edited and read by one person.
That is a weird take and after that I sort of just skimmed the rest. Yes building the datacenter is a huge expense, but you need to replace the hardware to stay competitive, and it doesn't last forever. Previously companies ran three year cycles, now five seems to be the standard. If hardware prices come back down, then.... maybe, if not those refresh cycles are going to put some datacenters out of business.
Another issue is that while e.g. Anthropic and OpenAI seems to be interested in building their own datacenters, they are mostly just renting capacity. If they can shop around, then they can get the price down, but there's a base price, due to power, hardware and staff.
The author also suggest that there's still capital to raise... no there's not. Anthropic is going for an IPO, that means institutional investors, not venture capital or borrowing and Anthropic, CoreWeave and Oracle are rated as junk. That doesn't mean that they can't raise capital exactly, but it will be very expensive and the interest payments needs to be factored into the price of the products.
Some hosting providers are known for not caring about the activities of their customers, but in general, how would I know that the other tenants are scum?
I have access to a fairly large dataset of known bad hosting providers, but most people don't. So how do you tell the difference between getting a good deal and "this is cheap because it's sketchy as all hell".
We also don't block all of Vietnam, or all Jio customers in India, even if we know that 20% of bot traffic is from those two alone.
A number of websites are starting to require an email in the user-agent header, as a way to identify your traffic specifically. This is in many cases done to be able to better identify scrapers and either apply rate-limits or contact the owner and ask them to cool down. (Generic browser user-agents will obviously be exempted from providing an email address).
The weird part doesn't seem to be that Claude Code would add an email address to a user-agent header, actually as a site operator that feels like a nice thing to do. The weird part is sites harvesting the emails and sending them spam.
Claude Code is actually doing, perhaps not the right thing, but it is being slightly more respectful towards website operators, in letting them know who is responsible for the agent traffic.
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
> but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.
If it turns out that the people behind social media, and smart phones, don't care about e.g. peoples mental well being, then why would I trust that the people behind AI wouldn't behave the same way?
Cars are actually a great historical example of this. The car was invented, and people saw the benefits, but some parts of the world went so car-mad that they started bulldozing entire neighbourhoods and deprioritising anything that wasn't a car. Decades later, you see the knock-on effects of those decisions, with high traffic levels, pollution, people spending hours in cars because it's the only way to go shopping, see friends, or take the kids to school. Places that took a more cautious approach still gained the benefit of cars, but kept the choice and quality of life for others
There are people whose livelihood depends on them not getting it. There are people whose worldviews depend on them not getting it. Denial is a part of grieving. There are also people who like to troll the ai maniacs because ai maniacs are easy to troll. It has been curious to me to see all this as well, smart people who seem oblivious feels strange.
AI has an extremely high capital barrier to research and development. Getting that initial capital required promises of a world-changing technology. A few companies with lots of investment were able to develop the technology and roll their own models. The business case for this R&D was that providing the models to paying users would amortize the R&D costs. But, promising disruption was the first step to getting the money.
Then the Chinese came along and showed that you can take those model weights, repackage them as open weights, and undercut the business model. Suddenly Dario et al. start talking about how dangerous AI is and how it needs to be regulated right now, before someone's abliterated open-weight Chinese-provided business-undercutting model destroys the world. They want their walled gardens regulated into being the only game available.
Both stages of the game—raising capital and regulatory capture—require hyping up the disruptive aspect of AI.
Didn't Dario leave OpenAI to start Anthropic specifically because he was worried about the risks and didn't think OpenAI was taking them seriously enough?
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
The US has for decades invested into charities and human development around the worlds .
> deregulation at all cost
The US has regulations. The Constitution of the United States is the oldest and longest-standing written and codified national constitution in force in the world.
It’s precisely because the corporate mindset of “profit before all else” that certain individuals become fabulously wealthy, enabling their philanthropy.
The Sackler family of opioid infamy were huge philanthropists. Not saying that the AI companies are like Purdue Pharma, but clearly philanthropy does not compensate for societal damage.
It's only cognitive dissonance if you look at the "US as a whole".
Americans can be charitable individually, but that does not contradict the reality that US corporate and institutional structures operate on a strict "profit above all else" mandate.
> Americans and the US as a whole are the most charitable in the world.
Yeah because it includes lobbies and turbo billionaire pastors from texas using "charity" money to buy rolls royces as well as other highly retarded """charities""" that aren't even legal in the rest of the developed world
You have senators and wannabe senators quoting the bible on TV as their guiding star as we speak right now, give me a break
On top of that you can separate the church and the state all you want, when people like Thiel show up it doesn't matter, these people are deeply troubled, if they were muslim and saying 1/10th of what he says you'd send them to guantanamo
Better than in the hands of a rabbi or ayatollah or christian extremist or various others. I feel simply profit minded people are one of the less dangerous hands in which powerful tools can exist.
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
You really can't think of anything? How about giving a fleet of AI drones a large supply of bullets and telling them to go kill everybody in some region or of some group.
Widespread mass realtime facial recognition usage is allowing governments to target and repress subsets of the population with an unprecedented scale and efficiency. ICE are flying facial recognition drones over sidewalks in downtown LA. Dissdents are having their facial geometries entered into secret databases simply based on political opinions.
Any new technology throughout history has first been used for war and genocide.
How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.
They literally targeted people for their past opinions, how they looked, etc. which you can now determine and process at break neck speed, you can even remove humans from the loop and have the machine determine who's an undesirable or not.
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
Well, are you thinking nation states? Then you kinda also need a population that is cool with a sort of blitzkrieg being waged "on their behalf" against the rest of the world (and they should think twice if that machinery wouldn't also logically end up being turned onto them). Otherwise, what's the race for, unless to use it offensively? So you either subjugate or eradicate the world, and then what? Who wants that?
Political power is when people act in concert, when leaders have people behind them because those people agree with how they're being lead etc., it's not something that happens unilaterally. You can physically force people, but for what when you can have robots that you don't have to force?
So that kind of leaves wiping out nearly all humans, for a few hundred thousand remaining people to get served by robots and play golf. Which I would say is what we're kinda on track for. But that wouldn't be a race between nations, that race would just be pretend to get the tech online; it would rather be a race between all humans and a bunch of sick people.
I'm thinking of nation states, but I fear you have a very limited view of how power is created and yielded.
AI doesn't have to be used offensively to give a nation power. The machines can already design new software at the drop of the hat, it will at some point become capable of designing new consumer and industrial products. Putting a tireless intelligence at the task of looking for new optimizations in your industrial processes and factories will eventually give you economic dominance, one way or another.
Offensive uses don't need to be as extreme as you imagine them. We're already starting to see the effects of having meticulous automated hackers. Instead of a blitzkrieg, imagine your foes suffer from a constant stream of industrial accidents, unfortunate events and information leaks that can never quite be traced to your nation.
Finally, political power itself doesn't need to come from people acting in concert. There's always been wheeling, dealing, intrigue and bureaucracy. Once AI is sufficiently versed in legal matters, it'd allow political interest groups to overwhelm governments and private enterprises with objections and appeals, paralyzing their ability to carry out their agenda until they make concessions.
The possibilities of using intelligence for acquiring more power are literally endless..
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
Which job are worth doing and which ones you'd rather the AI do?
AI should enable things that were previously impossible because of time constraints.
If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times.
For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
At some stage we might need to decouple income from traditional jobs. Bringing gladiatorial combat would be helpful. People can't stand a life of comfort. It will both probably become attractive to watch and a source of living for all, and it will solve the devils that emerge in idle heads. I am not joking, just look at the state of the Western world just from some basic material comforts having been fulfilled for a few decades.
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
The thing I keep thinking about with this sort of behavior is how Zoom became big in video conferencing. They had good features, yes, and COVID obviously gave every player in that space a big opportunity, but Zoom distinguished themselves in great part by doing every underhanded shady thing they could to make their app 'frictionless', privacy-invading, data-grabbing, and generally preyed on end users' lack of awareness. They did not keep doing all of those things, as far as I know – they got called out a lot but I have steered clear of them since my initial experience with them so I don't know how much they cleaned up their act. But taking the most aggressive and least ethical approach certainly seemed to have helped them get to the top at a very competitive time, and once there they could probably afford to stop doing the most egregious things.
It seems obvious that this same dynamic is in play with the frontier AI companies now. They know if they win the race to the they'll be forgiven their sins, or anyway they'll be beyond the reach of effective punishment.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations
Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.
It's sorta crazy to think how many years ago that was now.
> These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong
They certainly don't care when a model hallucinates or an agent does something it shouldn't be doing. It's your fault and they charge you for the tokens.
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money.
Can someone who thinks this about Altman explain to me why he has repeatedly declined to take any equity in OpenAI? I genuinely want to understand your theory of mind here. Do you think he's actually in it for the power, and for some reason thought taking equity would undermine his effort at accumulating power? It just makes no sense to me.
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
I really don't like the excuses that are inherent to the "well it was inevitable" line of thinking. Oppenheimer was still responsible for the bomb despite it being "inevitable" in a similar way.
Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.
Both nuclear bombs and AI are not inevitable. Technology depends not just on knowledge, but on infrastructure. Building and maintaining nuclear bombs requires enormous amounts of resources, and it's a conscious decision to continue spending those resources to maintain nuclear capability. Similarly, developing AI requires enormous resources, as you can see from the massive investments in hardware and data centers. If the economic foundation for it doesn't exist, you're not going to get more powerful AI.
The models we currently have aren't going away, and especially the ones small enough to run at home will continue to be used, but if these data center investments end up bankrupting Big Tech, I don't think we're ever going to see anything beyond that. Not for a long time, at least.
Does developing AI _require_ enormous resources, or is it using enormous resources right now because the people with the capital are throwing it at the AI companies hoping for a massive payout in the future?
My guess would be that once the money runs out, things slow down, but incremental progress continues to be made. A focus on smaller models will improve their capabilities. We’ll get better at driving them. It’ll be used in places where it truly makes sense, instead of anywhere & everywhere just because it’s easy.
I don’t actually know what the financials are, but even without the effective subsidy from VC money, I have to believe there are things AI’s doing today that’s way cheaper than what it would cost to have a human doing equivalent work. And that dynamic will drive further innovation in the space for a long time.
What does this actually mean? In what sense are people using this term. In the most strictly literal sense then sure, nothing is inevitable, because all paths are open.
But in the practical sense, I am quite happy to use the term inevitable for many technologies. I think if ASI is possible (and I think it is), then it is also inevitable. I would happily work towards developing ASI technology, because I believe it will be ultimately good for the world. But it sounds like a lot of people want to lock me up or even kill me if I did that.
I have also not been able to lean towards a side in inevitable v/s responsible.
1. Is it both inevitable and still they are responsible?
2. Or it was inevitable and they were just an instrument to make it happen.
3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?
I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?
If it is 3 or 1, then why aren't we holding them accountable?
"Inevitable" means "same technological progress would be achieved anyway, even if these people have made less damaging choices".
So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.
They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared
I am not sure what gripe you have with the labs, but I think if you are worried about AI then you got shockingly lucky with the lab leadership. My biggest gripe has been the campaign of negativity against ai.
Sorry, yeah I don't trust Altman and Musk, the companies they run inherit that. Their public face betrays an almost cartoonish evil, it's sad people can't see that.
Other labs seem to be working the exact same angles as them, but just have the good sense to not be so open about it. It's not the "lab" people have a problem with, it's their leadership. These people cannot be trusted to make the right decisions for society with this technology.
It's hard not to be when people at those labs talk about the probability of AI killing us all, and if it doesn't do that, then the massive disruption to the labor market that's coming. Combine that with the techno feudalist ideology of leadership.
So yeah, I'm negative against their development of AI along with the general public, and for good reason it would seem.
Also, I don't want super intelligence. I want humans to remain in control. Why are they trying to build something to replace humanity? Why don't we got a vote in this? All 8 billion of us not developing frontier models with the expressed goal of achieving AGI followed closely by ASI.
We didn't get a say in whether we should bet our entire economy on building out infrastructure for half a dozen guys to get rich from. No say in the way they are backdooring around the hamstrung EPA to install greenhouse gas machines to power these things. I think the pushback makes sense.
It's a pity because as computer scientists this should have been such a fun technological moment for us. What a cool thing to experiment and explore new open doors with. But instead it was immediately the driver of mass job loss, societal upheaval, and a double digit percent chance of ending humanity, espoused by the AI lords themselves.
I don't think you're anywhere close to a voting majority. Rather quite a small minority of the human population would want such a thing is my guess. It may seem otherwise in your bubble though.
Do that without dragging the rest of us into it then. I will continue to vote to block and advocate against such a thing on a societal level. It should not be thrust upon humanity without our consent.
Why the fuck should your fucking opinion matter to what I or someone else does to themselves and in their private property? Yours is the same logic as the anti abortionists. Its not "society" or "humanity"'s concern what medical treatments a person takes for themselves.
> if you are worried about AI then you got shockingly lucky with the lab leadership.
You must be joking. Altman is a sociopath, Amodei is either too stupid or too evil to stop working on tech he claims will have a 10% chance of destroying humanity (and in either case he should be imprisoned to protect the rest of us from him), and Musk is a man-child who will lash out at anyone and everything who dares to speak ill of him. There's absolutely nothing "lucky" about having these jokers in charge of stuff.
Other than Musk, the other two seem to me overall quite good stewards. Could have been much worse, and indeed hard to find someone better. How do you think someone could have been a better steward? There are 8 billion people in the world, almost all of them believe in absurd and mutually incompatible imaginary tales, its a very hard thing to create an information agent that doesn't offend a big fraction of the world.
I think it's pretty trivial to imagine ways in which they could've done better. For example, if you genuinely believe some tech has a 10% chance to destroy everyone alive, stop fucking developing it. There is no scenario where it is remotely acceptable to pursue such a risky goal. Amodei is either too stupid to see that, in which case he can't be trusted with anything sharper than a butter knife, or he's too evil to care about the risk, in which case he can't be trusted with anything at all. Or, more likely, he's lying to hype up his company, in which case he's a fraudster.
It's similarly easy to come up with "what Altman should do better". The dude is a genuine sociopath in charge of a company trying to make everyone's lives worse off in order to enrich him and his investors. It doesn't get much worse behaved than that.
If you impute its bog standard marketing/hyping, its the least evil and most mundane possible things you can expect in owners of such powerful tech. Your imagination is weak. Imagine if the owner was a Christian nationalist or Muslim fundamentalist as some examples.
The Oppenheimer comparison is very apt I think. Even if the science was inevitable, the decision to use the output (twice) in arguably unnecessary circumstances was not
The other difference is that Oppenheimer at least felt terror and remorse. The AI bros are saying "what we built might end the world but, lol, whatever."
Just to flip it around on you, what do you think would happen if Anthropic and OpenAI agreed to stop working on AI altogether?
I'm not arguing about the consequences or responsibility. I'm asking which humans would be capable of stopping the development of AI at this point in history.
The comments on this are interesting. There are a lot of assumptions from different people speaking about humanity as a single entity that decides how to operate as one. A lot of others act as though it’s really easy to get world governments to all agree to not develop a technology through treaties and agreements. One person below is arguing with me by saying that we “just” need to get China and the US to agree to halt AI and the problem is solved for the whole world.
It’s difficult to understand where this all is coming from because it’s so strangely disconnected from history. It’s not even consistent with current world events. The US is in a war with Iran over nuclear proliferation while also worrying about North Korea’s nuclear program. China and the US are in a trade war and Taiwan is under threat due to semiconductor capabilities (among other things, obviously). Open weight models allow anyone in any country to do AI work on their laptop. The top story on HN yesterday was a small team of academics making some cool discoveries on improving LLMs without frontier lab level compute.
How anyone can see all of this, in addition to decades of history of attempted nuclear diplomacy and wars, and posit that the whole problem is easily solved by simple agreements is confusing to me. The confidence with which these alternatives are being proposed and the unwillingness to acknowledge second order effects (like AI researchers moving to a different country and continuing to work) is baffling.
8 billion people. Almost all of them aside from a negligible fraction believe in at a fundamental level completely mutually incompatible fairy tales, a single digit or so fraction of them are killing each other over these tales. And some people think the world over would simply get on one page over AI?
hn is full of technical people that like to believe that they behave logically (they don't), and that others would change their behavior if the logic was explained to them (they won't).
But yeah, there are a lot of comments from genie-rebottlers that are not very grounded in reality. They're functionally equivalent to saying "hey, everyone in the world, let's not actually use electricity after all" in 1905.
People don't understand the prisoners dilemma, or they think it is some kind of psychological pseudo science trumped up to make fun pop-sci articles for social media.
Was it? We nearly destroyed ourselves with nuclear weapons several times in the last century. I don't think it was a good call just because we've gotten lucky so far.
Sort of, although didn't Von Neumann package it slightly differently - less personal (someone else) and more about the evolution of science and separation of roles of scientist and politician... basically science is going to happen, and it's not for people like us (scientists) to be worry about what happens as a result.
Still seems a moral cop-out, and while true, the timeline would certainly have changed, if not the inevitability, if key figures such as himself had opted out.
The development lead to it being dropped on two cities followed by an arms race that put global civilization at permanent risk, with several incidents already that could have launched a nuclear war between superpowers.
It's likely stopped lots of large direct wars between the biggest nation states. Instead it's much smaller proxy wars.
---
Simply put humans are at risk of any power scaling. We exist safely in a very tiny temperature and energy environment and the rest of the processes in the universe operate widely out of that strict safe area.
It's also a catch-22. There are large rocks in space that will eventually end us unless we do something about that. But having the power to do something about that gives us the same power to also end everyone on the earth.
After the war, it turned out that the Nazis weren't even working on a nuclear bomb. So had the US not done it, nobody might have. Other nuclear powers did it only because the US had them and had shown that it was possible.
> Nazi Germany undertook several research programs relating to nuclear technology, including nuclear weapons and nuclear reactors, before and during World War II. [...] The first effort started in April 1939, just months after the discovery of nuclear fission in Berlin in December 1938, but ended shortly ahead of the September 1939 German invasion of Poland, for which many German physicists were drafted into the Wehrmacht. A second effort under the administrative purview of the Wehrmacht's Heereswaffenamt began on September 1, 1939, the day of the invasion of Poland. The program eventually expanded into three main efforts: Uranmaschine (nuclear reactor) development, uranium and heavy water production, and uranium isotope separation. Eventually, the German military determined that nuclear fission would not contribute significantly to the war, and in January 1942 the Heereswaffenamt turned the program over to the Reich Research Council (Reichsforschungsrat) while continuing to fund the activity.
> The program was split up among nine major institutes where the directors dominated research and set their own objectives. Subsequently, the number of scientists working on applied nuclear fission began to diminish as many researchers applied their talents to more pressing wartime demands.
> [...] Heisenberg said that in 1939, the physicists at this second meeting concluded that "in principle atomic bombs could be made, but they also emphasized that it would take a number of years",[24] certainly not less than five. He also said that he "didn't report it to the Führer until two weeks later and very casually because I did not want the Führer to get so interested that he would order great efforts immediately to make the atomic bomb. Speer felt it was better that the whole thing should be dropped and the Führer also reacted that way."
> [...] Speer states that the project to develop the atom bomb was scuttled in the autumn of 1942. Though the scientific solution was there, it would have taken all of Germany's production resources to produce a bomb, and then no sooner than 1947.[39] Development did continue with a "uranium motor" for the navy and development of a German cyclotron. However, by the summer of 1943, Speer released the remaining 1200 metric tons of uranium stock for the production of solid-core ammunition.
There are ways of calling this "weren't working on a nuclear bomb", but it seems to me like "they wanted to, were moving in that direction, and would have done it if they had the resources and weren't distracted by shorter-term considerations".
Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.
Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]
And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.
> Humanity's strategy so far has been to pull marbles out of the bag as fast as we can.
> But why are we rushing so much?
Talking about these things with elaborate analogies only obscuring the point you already stumbled into: Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.
Talking about humanity like one entity that operates together misses the point: There are billions of people in the world and some of them are going to discover new things.
There is no general counsel where humanity gets together and decides what humanity works on. There is no singular bag with a singular “humanity” pulling marbles out one at a time.
The analogy is too broken to be useful for the discussion.
There’s a point at which logic/reason/wisdom overrides our curiosity, and there are quite literally multi-nation groups that collaborate on matters of global importance.
There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.
> There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.
You’re speaking of “we” as if the whole world was operating in unison, while there’s a war going on in Iran over nuclear disagreements.
The nuclear situation only reached stability after multiple countries fully developed it. I don’t see the parallels to halting AI.
It's hard for me to understand how the parallels wouldn't be abundantly clear.
The nuclear stabilization that occurred was the result of a collective understanding of the existential risk it posed leading to anti-proliferation agreements and superpowers going to great lengths to prevent nations deemed as unstable from obtaining such technology.
Some portion of the global community may collectively discover/decide that the risk AI presents is worthy of anti-proliferation treaties and wars may be fought over it.
The main gap in the comparison is that we have a much clearer understanding of the risks of unchecked nuclear weapons use than we do about what current and future AI tech might or might not unleash.
The underlying point is that we have historically placed limits on our curiosity when we deem certain paths of exploration existentially dangerous. It has yet to be seen if AI will reach those levels of existential risk, but if they do, there is plenty of precedent for stepping on the brakes.
And countries still violate these treaties. See Iran and NK.
Also the bio/DNA ones are quickly falling apart as the technology to do these kinds of things on a small scale are readily available. To the point where we're finding illegal wet labs in the US.
Nuclear was more of an exception because of many of difficulties in purifying stuff for weapons and its detectability.
AI is a much, much, much harder problem is it uses the already existing infrastructure as a base. Unless we regulate, monitor, and ban all private usage of technology AI can be developed at a small scale.
Is your argument essentially that because global treaties haven't been universally effective in all cases, we should just YOLO AI even if it has the potential to cause great harm?
It's not about merely knowing how to build a "badthing". It's about knowing why not to.
With AI, the point is who uses it for what.
Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?
And this sort of blanket argument is how you get bad science on the order of Prison experiments to Nazi scientists. Yes, academic research that pushes boundaries can be a $badthing
we can outlaw those kinds of uses, im kinda split rn, most people arent happier with AI but it is empowering so many people, especially around obtuse labor laws etc, it is a double edged sword squared
You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.
Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.
The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.
With AI, that reality gap will swallow whatever is left.
The market is irrelevant. In the 28 centuries of recorded history, the human race has pursued power in it's various forms (political, military or economic), without regard to the "public good" (which is really, just a polite euphemism for the established order and those that benefit for it).
Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.
> You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
I operate within the logic of how the world actually works.
Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.
Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
You're acting like international treaties (or cartels) don't exist. Yes, these are structures where parties have incentives to defect, and we counter those by creating incentives not to, with checks and sanctions.
Additionally, you don't need all the leaders of the world to agree to anything, you only need US and China.
Additionally x2, if you agree with the common stance that China makes the vast majority of their progress by distilling US models, you don't even need them to agree to anything.
A ban on AI looks nothing like a ban on almost every other technology that has come before it.
You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.
>China makes the vast majority of their progress by distilling US models
Eh, this is a budgetary decision, not a technological one. Why spend billions when the US can spend billions and you can spend millions. China more than has the power to do what we're doing if they chose to do so, their biggest limitation is speed of making new compute, not money or brains.
> You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.
You know what, fuck it, that's exactly what I'm suggesting. Let's ban personal computers. The only consumer hardware allowed should be a HTTPS client directly burned on the chip that can only connect to government servers. Absolute all timer of a straw man. This is probably the last comment I'm writing on this site, this place has become unsufferable.
Believe it or not, there already are things you're not allowed to do with computers that you otherwise own, and we have ways of enforcing them other than trusted computing -- the threat of putting you in a cell for a few decades usually does the job. That's without getting into the fact that you're not going to train 1T-sized models on your laptop, and billion dollar GPU farms aren't precisely hard to keep track of.
>Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
"We" (Israel, probably with US assistance) assassinate "rogue" (outside of existing nuclear powers) nuclear scientists (e.g. from Iran). So to take this analogy to its conclusion you have to accept that part of the pact would be assassinating people for doing computer science research or building computer systems that will be used for AI.
Nuclear weapons come with the immense benefit that purifying uranium is so challenging, expensive, and difficult to hide that we don't even bother to protect the information on how to build a nuke. They also have a second benefit of being impossible to test without the whole world knowing you did.
No, the AI compute supply chain is actually quite similar to the nuclear supply chain. Building chips capable of training or large-scale inference is extraordinarily complex, expensive, has a large footprint, and has a few chokepoints (rare earths mining, HBM manufacturing, and ultraviolet lithography, where there is literally one company in the world capable of building the relevant outputs) where we in fact already have governance mechanisms for things like export controls.
> The "world" isn't just what humans currently make of it. You deny the possibility of alternatives for society based on what amounts to ignorance.
Ignorance of what? All of the previous successful attempts where the entire world reorganized itself into a single world government?
> Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".
I think you’re putting the cart before the horse. You’re assuming it’s necessary and then making another leap to assume that being necessary means it must also be possible.
Arguably humans cooperate even more than they compete. Examples of cooperation are easy to find throughout nature. So is parasitism, but humans generally find that loathsome as a human activity.
The argument from nature is a fallacy. Just because something is doesn't mean it ought to be.
It’s more than that. Scaling LLMs had a fundamental problem - that it took $10b+ of investment to become useful. To get over this huge activation hump, you had to hype the thing as world changing. If you sold it as autocomplete you would never raise the money to do even autocomplete.
That’s why the AI heads all have extremely dramatic narratives. They are the end product of a selection tournament for raising massive amounts of money pre-product.
>Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
You make it sound like this was a feasible option. This technology would exist regardless of how any one company decides to frame it. If OpenAI decided to go with a more conservative approach rather than just throwing ChatGPT out for the masses, then all it would take is for one person to realize they didn't take full advantage of this technology to just make their own ChatGPT.
>The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
You can criticize the current tech leaders all you want, but what you're describing is a fundamental dynamic of capitalism. You must choose the most profitable option, otherwise you have chosen the wrong option. If the current tech billionaires all vanished just as ChatGPT was released, they'd be quickly and easily replaced with roughly equivalent individuals who would effectively operate the same exact way. That's true because they're operating based on the mostly deterministic policy of profit maximization. They don't really get a say in it; they must follow the path towards maximizing expected future returns, otherwise they'll get canned by the investors who are seeking maximized expected future returns.
None of this relieves them of their personal ethical responsibilities either. I'm just pointing out that your comment might as well be describing an alternate reality where everyone lives peacefully with each other and nobody goes without what society can afford to offer them. It's equally as unrealistic.
I think this recent post about a 2017 AI safety risk review may give some solid historical context that many of the leaders do not want to be the next Oppenheimer and have taken steps overtime to prevent being in that role themselves personally and collaborating with others so no one is: https://news.ycombinator.com/item?id=50366387
The only leader I’m not aware of really taking a more active role in safety on the daily is Elon Musk, so I’d like to see evidence supporting what he is doing or not doing on that front if someone from SpaceX can share their experience. The evidence on the environmental front from SpaceX is very concerning and is disregarding the wellbeing of the world and local populations in my opinion.
Musk has recently shifted to openly advocating for full-speed accelarationism, claiming that being a "biological bootloader for AGI" is a success condition for humanity. He's pretty clearly not concerned with preserving anything about the current state of the world.
Not against that concept in theory if it managed to be executed properly. Gene editing and centaurism always seemed natural next steps to me. I am willing to treat the viewpoint of some people wishing to slowing down or temporarily stopping such work in the sense that we must do a broader work of psychological/brain engineering to get people used to comfort and stop creating devils in their head when they are comfortable, otherwise I don't have any issues with transhumanism itself.
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
Still $5B is insane, that would make Postman worth more than JetBlue. Airlines aren't exactly a good business to be in, but Postman is fairly easily replicated.
reply