Proprietary techniques like this are usually a good indication you’re missing something. In this case it sounds like you are missing appropriate validation of the issuer and/or token itself.
I want to support OAuth2, not OpenID so I don't rely on a JWT; I call the issuer's endpoint directly from my backend using their official domain name over HTTPS. I use the sub field to avoid re-allocation of usernames/emails but my point is that I don't trust it on its own; I couple it with the provider ID.
To make it universal, I had to keep complexity minimal and focus on the most supported protocol which is plain OAuth2.
