It's mostly a VRAM limitation at this point though, since even a 81x slowdown with something like Stable Diffusion would be perfectly acceptable to produce the final high-quality render.

Which is to say, it sounds like the solution is dedicated GPUs that focus on VRAM over speed.