In an ideal world, absolutely! It's a hard problem and there are many attempts to make that happen automatically including polyhedral optimization (Polly[1]) and tensor compiler libraries (XLA[2] and TVM[3]). I work on a project called LoopTool[4] which is researching ways to dramatically reduce the representations of the other projects to simplify optimization scope.

[1] https://polly.llvm.org

[2] https://www.tensorflow.org/xla

[3] https://tvm.apache.org

[4] https://github.com/facebookresearch/loop_tool

If they worked so well AMD would not be in such a bad position with their GPUs in ML. They would just need to compile to their arch.