Scaling isn't nearly everything though. Scaling is low hanging fruit, but there are real constraints that limit it. For starters, we have finite data and increasing the size of our data set is going to get progressively more expensive (new data is going to be more niche as well). Beyond that, running these models on device is way more compelling than using a service, and the best current RLHF tuned Llama models that run on consumer hardware are close enough to ChatGPT to be quite useful.

The race isn't to build the biggest model, the race is to build the model that uses data the most efficiently and produces the best results when constrained to parameter counters that can fit on consumer hardware.