Most PC and server hardware has FLOSS drivers (with proprietary firmware), even Qualcomm is upstreaming support for the new Snapdragon Elite (maybe it's made by a different team?).

I think phone SoCs are the odd ones out, which sadly doesn't mean they'll improve any time soon. Supporting an ABI for binary drivers in Linux might help phones, but it would give everyone else a chance to regress in their support, so I understand Linux kernel developers' position.