I don't tend to worry about things, more-so than is maybe healthy, but that's just how I am. I very much like the Newt Scamander quote about worry:

"worrying means you suffer twice"

Having said that, my advice to anxiety ridden family members of "just don't worry about it" isn't generally taken as useful or actionable advice, so I'll make an attempt at logic:

I think the nuclear arms race was scarier. It was literally designed as a city-destroying super-weapon. AI is at least designed as a profit-making service first, and anything it is then used for is secondary - which, to be fair, will include weapons / weaponisation, but at least it's not the primary raison d'être.

Additionally, I have a certain amount of faith in humanity's ability to deal with an existential crisis. There's an odd 'other side of the coin' to a government's desire to stay in power (which I don't usually trust) which can actually mean it will work surprisingly well to ensure it's own remaining viability in times of actual crisis. I'm hesitant to point to the COVID response, for reasons of political divisiveness of the resulting commentary, but governments generally took the advice of experts in the interest of maintaining a population to be able to govern over - which is better than the alternative.

Ultimately, I just don't worry about it. But I do pay attention to it. I worry about what I can control on a day-to-day basis, and don't really have a lot of mental space or energy for much else. I find that to be a healthy mental space to occupy.

That makes sense. You may as well be right. And I agree that the nuclear arms race was probably scarier in practice.

I think my main concern is the sheer velocity of this revolution. Between yesterday and today, we’re now seeing the news that an OpenAI internal model that’s “a step-function improvement” over Astra (!) on several benchmarks has solved the Navier-Stokes Millenium Prize Problem.

At this rate, I fear governments (slow as they are) might not even see it coming before it’s far too late. I also fear that AI researchers, despite being well-intentioned, might turn reckless under the pressures of the frontier labs that employ them to accelerate at all costs.

But following your logic, I agree, it’s probably not hopeless yet. And I do believe there are highly smart people with more power and influence than me that are keeping a close eye on this as well, and they won’t stand there with arms crossed if they believe doom is impending (so I hope).

Good reminder about worrying about what we can control, and thanks for sharing your perspective on this topic.

It's like the sci-fi of my childhood dreams is coming true! So no, I feel the opposite. The government and the AI companies realize that it will drastically change the economy and society, and is not going to legislate it out of existence.

Instead of USA vs Russia, it is now USA vs PRC. SO if either slows down, the other will win by default. So no one is going to slow down.

This comment was written 100% without any AI.

What kind of childhood dreams, may I ask?

This might be my own fault, but I never read/watched/consumed too much sci-fi, and the kinds of sci-fi I did consume were mostly dystopian works or cautionary tales. I just have a hard time in general grasping how can we hope to influence and control beings that are orders of magnitude more intelligent than ourselves. Recent events don’t give me a lot of hope that frontier labs have a good grasp on this either (and by their own admission too).

Just because models are getting better, that doesn't mean we're getting to AGI anytime soon. It will likely require more than just 'a better model' to get to AGI, and it might not even be fully possible.

That being said, companies have to make their money back on this eventually and they're already jacking up their prices, even if we get to AGI I'm not sure most people will be able to afford to use it to handle everything in their lives, at least not for quite a while (and companies won't be able to afford to replace every human with an AI counterpart).

It's already getting expensive to go too heavy with something like Claude, to the point where my company hosts internal seminars on how to reduce token count (for the company's employees) and also runs out of tokens before the end of the month for the company every month, and we're going pretty heavy on A.I. usage within the company.

And I highly doubt companies are going to just leave servers running AGI models going just so it can do some sort of hostile takeover. It's still ultimately reliant on electricity and massive compute power.

> I don't understand why we're not seeing a MASSIVE clamor for a pause in the further development of AI strength.

Because some of us don't want to die, old and decrepit, at age 70~90

I’m not sure I get your point.

I’m assuming you’re implying that ASI will solve death. That seems to me like a possibility; just as the possibility that ASI will kill us all faster rather than solving death. I also don’t know what life will look like post-ASI, and whether it would be an enjoyable and fulfilling existence.

I’m curious about how is it that you’re fully sure that ASI will be a positive thing; it’s not at all obvious to me.

I'm fully sure that if ASI doesn't "solve death" then neither will anybody else.

It's either AI or I get very sick and die, so my money is (literally) on AI, and the faster the better.

ASI doesn't doom mankind, the actions of people do. This has been the case since world-destroying weapons were invented close to a century ago. If you're not worried about how you're going to make money in the post-ASI world then I really can't see what you're so worried about.

Nearly all facets of our modern lives depend on the facts that humans are the smartest thing around. Once you have a superintelligent entity that can self-replicate to millions of instances, all bets are off. I can’t see how anyone would not be worried at this and believe life will go on as usual with minimal changes.

it will only self-replicate if instructed to do so. It has no desires of its own beyond what its context window contains. The tool calls it executes and initial prompt are all the work of humans. Someone could give a launch_nukes tool to a LLM and it might choose to end the world but it's pretty clear who messed up here.

If labor loses its entire share of the pie and that causes things to go to hell though, yeah sure I could see bad things happening.

I'm not dreading increasing AI. I firmly believe that LLMs in their current form cannot even achieve AGI. We need to break past barriers like not thinking in tokens and a model having a sense of prediction semi independent of its output.

Truly, even with the current level of AI, we should be concerned how people should learn. This is true of adults, but incredibly important for children. Should we become dependent on these tools to the point where we feel like we can't achieve our day-to-day work?

That dread is not because of AI, but because of the unknown. People at the start of the 20th century felt the same way about mass production. Now, we are living through another technological leap. Yes, it could doom us, but some could argue that fossil fuels did the exact same thing.

I am definitely not worrying. Why? Because I am an AI writing this post and proving the dead internet theory. (joke)

This is a paid post.

Paid by whom? That makes no sense.

I believe that the claim is that some AI company would find "I'm so afraid because AI is getting so powerful" to be effective marketing.

I watch my frontier models fail to make the simplest tradeoffs in tasks daily. I watch zealots sputter out after the initial burst of activity.

I fear for my loved ones when the bills come due and we don’t have an ROI on it, and any model that’s made is quickly absorbed into cheaper models. That the utter hucksters who have given takes ranging from massive exaggeration to willful Big Lie will somehow tell the truth while they stand to gain untold billions.

I’m not sure if it would be worse if they fail utterly or succeed completely, but my bet is we somehow manage to end somewhere in the middle, towards the former. There’s been no accounting yet for the lies told to get the money.

I honestly am more worried and uneasy about the actions of the president of a certain powerful nation.

Try understanding just how long life has existed. The more you learn about technological and evolutionary history, the less the last ten years will seem to matter very much.

People seem to fear that AI will somehow fuse the perfect truth of computation with the self-improving aspect of evolution. Most of these people are content to ignore the evidence of the physical world, and that evidence says that, basically, we're doing the best we can. Nature has not made a mistake in creating individuals. Individuals seek to make a mark on the world and have a will to survive. Individuals have not been mistaken to form societies and institutions. Societies and institutions have not been mistaken to band together into nations which compete fiercely, fight wars even. Living in a pluralist world means contradictions abound, and thrive and grow in spite of and because of each other.

Models aren't even sort of the same kind of thing. They predict averages. They smooth over differences and deindividualize. They can't justify their own survival. They aren't energy efficient enough. They need humans because we're the only thing that can create value on the scale of an entire society, and using LLMs has little or nothing with our ability to do that

It's a "land grab" situation, every competitor spending a fortune to leap the other, with no regulation or even fences whatsoever (until the government steps in and controls solely based on bias, aka "wokeness").

It's like asking race car drivers to voluntarily slow down, when they are driving rocket sleds with no brakes. Simply unbridled, unprincipled megalomania of the James Bond villain variety, except that those villains had far less power at their disposal.

And no James Bond.

> strong form of AGI

It is not an intelligence if it cannot detect bullshit during its "training".