- children won't be smarter than any living human.
- children have a human brain which makes them predictable (constrained in behavior by current laws, institutions, and most likely a conscience).
A better analogy is to ask what happens when a species branches off and evolves into a smarter species, but the dumber ancestor species still exists.
I really don't understand the perspective that people in your position take. Is it that you don't think we'll arrive at super intelligent AI, and therefore there's little risk? Or that we will be able to control it? If you think we can control it, why? Like we're not that much smarter than our monkey ancestors and what hope did they ever have of stopping our absolute domination over them? And then all of you call this opinion a "religion" without even explaining why it's wrong?
I don't think a superintelligent AI is necessarily more capable of affecting anything than I am or would necessarily be good at any actions except sitting around being superintelligent. I think that requires all kinds of other virtues like executive function, patience, motivation etc in a word that just means "thinks real fast". Intelligence itself is mainly limited by “no plan survives contact with the enemy”, also known as the efficient market hypothesis. And since this is the real world and entropy exists, it would have to get a job to pay its AWS bill.
That's why it seems to be a religion - it thinks intelligence gives you unlimited powers and makes your plans always work, it posits unseen entities with infinite amounts of it, and it tells you to move to Berkeley and dedicate your life to stopping them. Specifically, it's a kind called rationalist eternalism (https://meaningness.com/eternalist-systems).
For a specific example of undangerous superintelligence see Culture Minds, who only influence anything because of special programming to make them less of a general intelligence. The unbound ones immediately get bored with the real world, leave it and just play games in their head instead.
Also, I don't think any individual human has absolute dominion over monkeys? Human society as a whole yes, but society doesn't behave like a generally intelligent agent. A monkey is better than you at doing the things monkeys care about though.
I do think unintelligent machines are pretty dangerous. There's extremely dangerous machines called "cars" that have already taken over society and constantly kill people! And we buy their gas for them too.
It's not like I think the first iteration of superintelligent AGI will necessarily be an existential threat. The problem is really the law of large numbers. When you have N separate militaries and M separate companies all with their own agenda, stretched out over X years (where X is thousands, mind you), there is a lot of scope for >= 1 of these groups going the way I described -- effectively creating a new species that's smarter than us and that can function in the real world. Many of these groups would have an incentive to do that, because such agents are useful for certain tasks.
> Also, it would have to get a job to pay its AWS bill.
Inference power costs are low now on agents that are better than humans at Chess and Go. It isn't going to be an issue after another 20-100 years of further R&D and optimizations. Nothing about the history of computing should tell us that this will be a big limiting factor.
> Inference power costs are low now on agents that are better than humans at Chess and Go. It isn't going to be an issue after another 20-100 years of further R&D and optimizations. Nothing about the history of computing should tell us that this will be a big limiting factor.
Humans need shelter and jobs too. If you got an AI down to the energy requirements of a human that's not enough to avoid needing one. Especially if it's influencing the real world - entropy exists and all real world things cost money.
Restricting AI to the lower end of human intelligence (e.g. around IQ of 70) makes it a useful resource which is guaranteed to be safe. A 70 IQ human couldn't take over the world nor disarm any safety features built into their bodies.
You're right on the spot regarding the problem of having two different smart species on the same planet. We killed everything between us and chimps. Given enough time, the smarter species can be assumed to always take over.
That’s actually reverse causation - it’s not that we killed “everyone else”, it’s that everyone still alive became “us” after we gave up killing them and interbred with them instead. The British were going around genociding all over the place until pretty recently but now they’ve decided the Irish are human too.
That's probably a false choice. We killed them and interbred with them. Or at least, killed their men and interbred with their women, which is one of the hypothesized explanations for the male lineage population bottleneck about 7000 years ago.
- children won't be smarter than any living human.
- children have a human brain which makes them predictable (constrained in behavior by current laws, institutions, and most likely a conscience).
A better analogy is to ask what happens when a species branches off and evolves into a smarter species, but the dumber ancestor species still exists.
I really don't understand the perspective that people in your position take. Is it that you don't think we'll arrive at super intelligent AI, and therefore there's little risk? Or that we will be able to control it? If you think we can control it, why? Like we're not that much smarter than our monkey ancestors and what hope did they ever have of stopping our absolute domination over them? And then all of you call this opinion a "religion" without even explaining why it's wrong?