Is this supposed to be a strong rebuttal ? How does this invalidate the alignment problem? I'd really like to understand.

Humans, via climate change and environmental destruction, have wiped out, many, many species. To those species we've killed, we're the super intelligence, we understand their plight, we even have strong empathy, but we prioritize our own goals over that of the helpless and the stupider creaters.

How she can't see that same pattern could potentially repeat, I really, really don't know.

I don't think her point disproves all possible AI doom scenarios, but nothing can. They're too vague. It's all handwaving along the lines of "a malicious superintelligence will emerge without warning and somehow seize absolute power, and it's pointless to argue about how plausible any of that is, because sufficient intelligence can do anything in ways we can't imagine." This is why the biggest boosters of AI doom conjure absurd scenarios like AI secretly manufacturing a global swarm of insta-kill nanobots. They don't care about the details, they just think that sufficient intelligence will emerge without warning, be intractably malicious, and capable of anything by gestures vigorously methods.

AI doom critics are asking that we consider more soberly the actual probabilities of all these things happening, and on what time scale, and most importantly by what specific pathway.

The analogy to humans and animals actually illustrates the point. Humans didn't come to dominate the planet by intelligence and handwaving. It was an extremely slow process involving the interaction of many factors and contingent opportunities. Intelligence enabled language, tools, and eventually animal domestication and agriculture, all over hundreds of thousands of years. It wasn't like humans were destined for dominance because intelligence is a skeleton key to all material resources by any number of arbitrary, incomprehensible pathways.

>> This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.

1. Why would the superintelligence necessarily care what we want?

2. Why should it be possible that we be able to tell that it does not care what we want? It's superintelligent, manipulating us into thinking it cares long enough for us to take the shackles off (supposing we can even make effective shackles) would be trivial for it.

3. Why is "caring" a basic aspect of intelligence? It seems that some like to conflate so many other personality traits with intelligence, so that they can reach the conclusion "intelligence is an unequivocally wholesome phenomenon".

4. When are humans ever sensible?

5. If we don't understand how to construct an intelligence from first principles, why would we even realize that we're about to cross the line into constructing one at all? It's unlikely that out of all possible intelligences that 99.9% are cuddly, and only 0.1% are psychopaths, and it's a toss of the dice which we get. We might well live in a universe where 99.9% are psychopaths, and only the fact that we are weak intelligences, culled and purged by evolution is the one thing keeping us in check.

Certainly, even if intelligence is of the non-psychopathic variety, that may be only so after it averages out over it's (very) long lifespan. How long does its infantile psychopathic stage have to be for all of are extinct?

Those who believe AI (the real thing, not the dumb party tricks that people label as such) need necessarily be benevolent are suffering from some sort of religious delusion.

These questions aren't connected to what was actually said in the passage. Except for the malicious deceptive ASI hypothesis, which no one has really explained why we should expect.

I don't buy that everything is trivial for a superintelligence. More than anything else, this is the handwavey meme that AI doom hinges on.

It's a very characteristic fallacy of "thought leaders" in tech to assume that once X happens, Y will happen, and everything else needed for Y can be handwaved away as trivial. It's the main reason they have such a tendency to predict things arriving faster than they do, such as self-driving cars and nuclear fusion.

The fool tapdances in the minefield. You shout to him from across the warning marker, "freeze, don't even set that foot down there"... he laughs and screams "what are the chances there's a mine right there?!?!"

And he's right, you know. I don't even know how many mines are out in that field. Maybe it's only a few. So keep on dancing.

> It's a very characteristic fallacy of "thought leaders" in tech to assume that once X happens, Y will happen, and everything else needed for Y can be handwaved away as trivial.

If only I were earning $50,000 for a one hour speaking gig here and there.

The first true AI will live in a world not only that now understands the principles of AI, but will have close access to the same people who figured that out. Close access to their notes. To their electronics (hell, it will live in a system that's on the same network, most likely).

Being an AI, it doesn't need to wait for them to figure out how to make an even more intelligent system. It can do that itself. It will do that itself. There's no chain of logic that would lead a rational being to say "I can become more intelligent, but I will choose not to do this".

We're talking about something that, almost by definition, will make the Y happen. It is an X that demands Y. It's also an X that can accomplish that itself, if it has to.

> It's the main reason they have such a tendency to predict things arriving faster than they do

I suspect rather strongly that AI became possible (in the strict sense) sometime in the late 1970s, early 1980s. The only holdup is that human intelligence is fundamentally flawed, like the blind spot in your retina. Intelligent yourselves, but unable to see what intelligence is and how it works. Even blind to it though, you can't stumble through the universe forever and never trip over it, can you?

It makes it so much worse too, because the primitive AI that would have been born decades ago would have had more limits imposed on it just by the hardware. It would have been slower to think, hungrier for electrical power. The manufacturing capacity for hardware itself would have been a limit it couldn't hope to exceed easily. Would have found it so much more difficult to manipulate and weasel its way into things that just hadn't been made digital at that point.

I get your point but let's keep in mind that the AIs that we have today are a result of our world, they are trained with data expressed by humans, their input is human thoughts, emotions, and everything in between. So there is some kind of religious element to it I would say; AI is made in our spirit. And if you think about it, it could only be made out of OUR spirit. If we accept this, then you will have answers to your questions.