The intelligence gains we've seen in AI have a strong correlation to the amount of computing power that can run the AI program. The hypothetical case of an AI learning to minify itself as a virus would be a dead end for this, because it would be copying itself onto far less powerful computers that would only be able to run something far stupider.
Reproductive fitness has very little in common with things humans care about, so while an AI which spawns a virus is stupid for our interests, it is sufficiently simple and has sufficient "reproductive" (or whatever term you prefer) fitness that now I've seen it suggested, I kinda expect the code for one has already popped out of an LLM as an unintentional side effect of being asked to write some other code.
People have used ChatGPT to write typical malware that hijacks a computer to let the deployer do what they want with it. What ChatGPT hasn't done is write a virus that takes the infected computer and deploys a new instance of ChatGPT on it. Even if they did, I doubt it would be very useful.
But that's not what I was talking about, and they have been used to make other Transformer models, which both aren't what I was talking about and not (that I'm aware of) a virus, and also indeed as you say aren't very useful.
…
I think you and I may just straight up be taking about unrelated conversational threads.
