>> But now, suddenly, it's highly likely the problem will be cracked within a decade, and it will be done by corporate R&D teams, by means of scaling up the transformer architectures.
I'll counter that. In the end we need AI that can do training AND inference on edge devices out in the real world. A good (and possibly profitable) example would be robotic pets that can learn (even to understand words) and interact with their owners like real animals, but don't need to go to the vet or eat and poop. Big companies relying on huge compute resources are not even aiming at this type of thing. They're too busy using their "scale" to even bother looking at smaller but interesting methods or solutions.
That's fair, and I'd agree in any other case, but This Time Is Different™. If (when) someone cracks this problem, they will be able to apply compute to every other problem. Some marketer convinces some C-suite that the world needs smart robotic pets? Why, here's access to ChatGPT-8, make it so. Just don't burn more than half a billion dollars in a week on it.
That's of course if, by this point, we aren't in a middle of a futile scramble to avoid getting extincted by ChatGPT-7 that someone left in self-play mode and forgot to turn off before going on vacation.
Point being, general AI is general. Even at extreme expenditure of resources, the closer the corporations get to it, the more problems they can put it to - including, eventually, the problem of optimizing itself. Already today people are using current-gen models to assist in developing next-gen models; this trend will only continue, until at some point you'll be able to let the model self-improve, mostly unsupervised. I imagine the compute costs per AI value delivered will drop like a stone then.
I'll counter that. In the end we need AI that can do training AND inference on edge devices out in the real world. A good (and possibly profitable) example would be robotic pets that can learn (even to understand words) and interact with their owners like real animals, but don't need to go to the vet or eat and poop. Big companies relying on huge compute resources are not even aiming at this type of thing. They're too busy using their "scale" to even bother looking at smaller but interesting methods or solutions.