4 bit has always been accepted as the quality floor that retains most of the models capability. Going beyond that brings problems. It doesn't make models totally unusable, but my contention is the quant providers are intentionally not showing an accurate picture, and then everyone is glazing 3-bit quants. They are what they are, but it'd be nice if we had a more accurate shared picture of it rather than just hype and vaguery.
I agree with him. And I'll gladly donate my 128GB of DDR5 if it means OpenAI and Anthropic will be nationalized to ensure equal access to resources. Central planning will surely create the most efficient system
reply