Accelerate on the M1 is ridiculously fast (thanks to its special core set and specific instructions).

Some benchmarks I've done has it beating out CUDA on my RTX 2070. I have to got a proper gflops number though

It's going to absolutely blow this away. Here are some of my single precision GEMM benchmarks for the M1 and M1 Pro:

https://github.com/danieldk/gemm-benchmark#1-to-16-threads

tl;dr, the M1 can do ~1300 GFLOP/s and the M1 Pro up to ~2700GFLOP/s.

On the vanilla M1, that's 28 times faster than the best result in the post.

The difference (besides years of optimizing linear algebra libraries) is that Accelerate uses the AMX matrix multiplication co-processors through Apple's proprietary instructions.