That's 1.6 Gbit/s. Looking up random Intel Xeon, specs, the memory bandwidth from the CPU to memory is 100 Gbytes/s. That's 1000X more than what you are describing.
You only need to pass a bin file to program an FPGA - it's like 20Mbits (hence a few msec).
Memory interfaces are not the same as the reconfiguration programming interface.
The memory interfaces (to external memory) are standardized - DDR3/4. Are you talking about to a L1 cache? Because all of the equivalent in an FPGA (BRAM, in Xilinx terminology) is accessible within a single clock cycle on an FPGA (assuming you can meet P&R/timing). Each DDR4 chip can do something like 20Gbit/sec, I can fit 5 of them on a KCU115 1924...
And that's external memory. Internally there is BRAM (and it's newer variants, which means you don't even have to go off chip).
That's 1.6 Gbit/s. Looking up random Intel Xeon, specs, the memory bandwidth from the CPU to memory is 100 Gbytes/s. That's 1000X more than what you are describing.