True, that's maybe the only case where there's an improvement. But it's a one-time cost (the browser can cache the compiled version), and switching to bytecode is not necessarily the only way to improve performance.
For example, a JS parser specially optimised for asm.js.
Note that cache is not always a solution. Cache is a terrible answer for first-time user experience. A lot of stuff on the web is loaded outside of the cache too for various reasons. Caches gets flushed.
My suspicion is that it will never be possible to parse asm.js as quickly as bytecode simply because it is structurally different. However, a quick search does suggest there are still some improvements possible in Firefox to speed things up somewhat.
