Can they afford to drop the prices without causing investor panic? Feels like Nvidia needs to constantly exceed all expectations to justify their share value. And if their share value drops, it might have a negative impact on their ability to finance their own customers, and if their customers run out of funding, they’re gonna have even more trouble shifting units.
Yeah. I for example use local inference for tagging up my documents that I self-host in a local Paperless instance, and I don't really feel like sending my medical invoices to clammy Sammy.
Not sure if it would even need to be a small Chinese model. Amazon would surely have the necessary hardware and infrastructure to run a bigger Chinese model with a low per-token cost too. They probably get their Sonnet at a discounted rate too since they do the inference "locally", but I imagine the savings for not having to pay Anthropic their share would still add up.
I don't the envisioned scenario is that a data centre company fails and Nvidia starts making RTX 5090s for consumers. Rather, one of these companies fails, their stock of enterprise GPUs gets flooded on the market, and suddenly the people/companies buying RTX 5090s for local inference can buy full fat enterprise GPUs, shifting demand.
If by enterprise you mean datacenter (which is what CoreWeave is), the "full fat" cards aren't as plug-and-play in a desktop PC as the consumer cards. They don't typically come with fans, and are meant to be cooled using some kind of holistic whole-rack (or larger) setup. Hobbyists do sometimes buy datacenter GPUs secondhand (and I have considered it, for the price/performance ratio) but it's kind of a lot of work and they tend to run hot.
There's probably still plenty of demand for that. At least someone in the executive team was floating around the idea of buying up some Nvidia cards to run AI models for the entire company somewhere locally. I don't think they'd be opposed to buying up data centre GPUs without fans if they suddenly flooded onto the market, even if that meant getting a rack chassis and a bunch of noisy server fans.
I've also seen someone sell their old fanless Nvidia cards on the local marketplace. There are definitely consumers running these things in their basements.
If a neocloud fails, it makes more sense to recapitalize it and leave the GPUs in place. If you sell the GPUs nobody will have enough power to run them.
Would it? First of all, all current AI users still need memory in their devices. You can't use ChatGPT with pen and paper. There would always be a consumer demand for memory, even without local inference. Secondly, it'd drastically decrease the demand for HBM, which has shit yields when compared to conventional memory. It's also questionable as to how much of the produced HBM is actually in use and not sitting in a warehouse.
reply