Since at least Windows Server 2008 R2 in Server Core mode you can uninstall 32-bit compatibility:

https://msdn.microsoft.com/en-us/library/windows/desktop/dd3...

Additionally Nano server does not have it.

Not sure if it is yet possible to remove from the desktop yet, but I'm doubtful.