I would think USB would be more of a OS issue, though. If a USB driver came as a blob, my Talos II obviously couldn't run it, but otherwise pretty much all USB stuff just works if I have source code for it (Fedora). Page size can ruin your day for some devices -- I have to hack FireWire, for example, because it assumes a 4K page size and Fedora uses 64K pages -- but so far no issues with USB or AMD GPUs.
Optimization is a bigger issue, though autovectorization in compilers is making it less of a problem than it used to be, as well as us nerd pioneers getting things upstreamed.
Although it's one of the "different" things, along with memory model, the page size is the same for Fedora aarch64 if I remember the discussion right.
Is POWER9 compilation generally poor compared with other targets? It didn't seem so to me, apart from a pathological case on one benchmark set which IBM addressed swiftly. They've been supporting GCC for rather a long time.
That used to be the case until https://www.spinics.net/linux/fedora/fedora-kernel/msg12805.... which put aarch64 to 4K. I argue there should be a workstation ppc64le spin that does this, but I understand why Fedora doesn't want to put the releng resources towards a niche audience.
I agree IBM's autovector stuff is quite good in gcc but there's no substitute for hand-rolled assembly sometimes.
Optimization is a bigger issue, though autovectorization in compilers is making it less of a problem than it used to be, as well as us nerd pioneers getting things upstreamed.