You've fantasized a situation and drawn conclusions without any real guess as to the plausibility of the situation.
We're supposing armed forces are using AI very heavily, but not so heavily to defer to it. They think it's valuable, but it made a bad choice in this case. But they also think it's not such an obviously bad choice that the opponent will also overrule it.
So first of all, we're assuming a hypothetical where we, as humans, also judge a first strike to possibly be the right course of action. After all, the hypothetical assumes we will be worried that other humans will think that. This is not a new concern. It's not good, but it's not new, and it doesn't apply to situations where a first strike is obviously a bad choice for both parties.
But then secondly, we're presupposing the AI is pretty complex and valuable and usually gets things right. We wouldn't be almost-deferring to it if not, and we wouldn't be worried the opponent defers to it. And we would certainly have informed the AI that the opponent is also using AI. And while I'm very hesitant to reason "it's unlikely the AI would make this recommendation to begin with", it does seem unlikely it would do so in any event where it's demonstrated such strong capabilities that we've entrusted it this heavily. We're essentially presupposing it doesn't do that.
I don't disagree with that, but then stating something as "the biggest worry" doesn't actually seem to communicate that, especially if it's a not-very-plausible situation you proposed.
We're supposing armed forces are using AI very heavily, but not so heavily to defer to it. They think it's valuable, but it made a bad choice in this case. But they also think it's not such an obviously bad choice that the opponent will also overrule it.
So first of all, we're assuming a hypothetical where we, as humans, also judge a first strike to possibly be the right course of action. After all, the hypothetical assumes we will be worried that other humans will think that. This is not a new concern. It's not good, but it's not new, and it doesn't apply to situations where a first strike is obviously a bad choice for both parties.
But then secondly, we're presupposing the AI is pretty complex and valuable and usually gets things right. We wouldn't be almost-deferring to it if not, and we wouldn't be worried the opponent defers to it. And we would certainly have informed the AI that the opponent is also using AI. And while I'm very hesitant to reason "it's unlikely the AI would make this recommendation to begin with", it does seem unlikely it would do so in any event where it's demonstrated such strong capabilities that we've entrusted it this heavily. We're essentially presupposing it doesn't do that.