I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
It's 2xTP 2xPP. Each system runs 2xTP cards, then split and pipelined. This works because each server is doing TP only over its local PCIe and only the split traffic, which is much less, goes over the 25Gbps RDMA link so there is virtually no penalty for doing it this way, but it also does not give any single user speedup. It does free up more room per card to easily host 8x262K KV in VRAM and the vision tower. So it is pretty nice and I am very happy with this setup. I mean, behold: https://stealthgoat.com/fnfl/pelicans6.html the evolution of the pelican :)
Thanks, super information. I am considering a TP4 PP2 system using 2 nodes with 4x 5060ti 16gb on each. Already have 6 such GPUs, with 4 of them running 27B at the moment. Hoping to get similar performance. Ideally with stock vLLM to keep on a well maintained standard software.
You should be fine with 5060 on vLLM. The support for V100 is limited but getting much better. You should be able to get good performance. TP4 will hurt across any sort of network link, but you can see, the 5060 has a lot of compute power so the delays won't hurt as much.
That should be solid. Just splitting / pipeline over the network link is not bad. Tp4 on PCIe you will feel it for single user and it might even reduce single stream performance but if you batch and have concurrent workloads it will give you a lot more throughput. It will give you a lot more room for context and weights though, which is nice :)
It is basically a gutted ggml/llama.cpp, but I just had the agents build highly optimized version for two specific model quants with zero consideration or compatibility for anything else.
https://github.com/bitexploder/fnfl-v100 -- I am still fixing the odd bug in it and it is really specific to ISTA IQ3_XXS and Unsloth UD4_K_XL in terms of what I have built. Some of the smaller benchmark data is out of date, like I think I got 32GB MoE cache V100 up to about 700-1000pps.
I have ended up using Argon more and more. If you keep it grounded it is a great model. Extremely thorough. A little slow. It feels like it gets things done better than Opus 5.5. It slowly churns through everything. Not necessarily quality but its adherence to goals seems better and it takes less prompting.
For cyber use cases I am absolutely blown away. Can’t say much more, but it is a serious model.
Just my opinions. I really recommend people take it seriously. For many jobs I just prefer it now. It is a new model so it needs time. Opus 5.5 is a sweet spot as well. I use both extensively.
Local AI already is really useful. This has been true since Qwen Flash Next and Qwen 3.8 27B rolled out. The struggle with exactly 64GB of unified RAM is that you end up having to stream the engram table from SSD because you will need almost all of your 64GB to run QFN. A 27B dense model still runs a little slow, even on an M1 Max Mac. However if you have even an older gaming GPU with 12-16GB of RAM and enough system RAM you can run this model (QFN) at a very respectable token rate and intelligence level. It is a Sonnet 5 class model you can run at home. There are new quantization techniques that make these models very good at agentic tasks.
And now if you stretch you can run DeepSeek Flash 4.1.
And if you were willing to roll up your sleeves you could build your own AI lab at home and run this model and models like GLM Flash 5.3 and DeepSeek Flash V4.1 at lower quants. The budget GPUs (like V100) are kind of not budget now, but still <5K to acquire 128GB of fast VRAM that can compete with a 5090 and actually has a lot more interesting options available IMO.
Can share a good way of running Qwen 3.8 27B (or similar) on 16GB VRAM? I've been trying but I end up with either a heavily quantized model or a small context or a slow speed.
Dense models are not well suited to this. Depends on your system, but most inference engines let you put the KV cache on the host. They should let you use "Q8" KV cache format as well which takes half the storage of bf16, the default format. You essentially are stuck at having to fit the entire dense model into VRAM because it must compute across the entire model every token. Without it being in VRAM it is slow. MoE models such as Qwen 35B with 3B active parameters are much friendlier to limited VRAM systems. This is why you see systems with 16GB of VRAM running something like DeepSeek Flash if they have a bit of spare system RAM. Usually not at any good token rate, but tolerable.
Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.
I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.
I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.
Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.
Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.
QFN Q4 K XL Unsloth is basically identical to the bf16 model. It’s KL divergence is very very small. It is competitive with the full model. Also I benchmarked iq3 xxs on Strata for 3 days against deepswe and it scored better than the published benchnarks for the model. If the gsq rco quant prettain tasks fit well it is a great model quant. My data shows it takes 20% more tokens basically than q4 xl. A4 xl is within noise on all my repeated task benchmarks over the public model serving and my own runs of q8. I published the deeswe results. You have to be very specific about model choice and know these models and their quirks.
This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
> QFN Q4 K XL Unsloth is basically identical to the bf16 model.
Yes, but like original poster I have 16GB of VRAM and 64GB of RAM, I was referring to non-Strata benchmarks :)
I did quickly run a couple things on Strata but haven't gotten around yet to benchmark bigger models/customize them with Strata, but will do rather soon. So far Strata/Swift IQ2_XS performed really well compared to non-Strata/Qwen3.8 quantized variants.
> This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
Yep, totally get it, and to be honest getting to run ablated models sometimes can also be useful.
> Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
My rig is... ~1.5k, maybe 3k if we count the CPU and GPU upgrade since 2019 :-D I really wanted to grab one of the latest mini-PCs (Halo, Spark, and other minor assemblers) for local setup, but to be honest I can't justify a 6k+ purchase for me and my wife alone. Perhaps in 3 years these mini-PCs will get to the ~2k ballpark (perhaps used), and although older by then, I might be able to run things 24/7 locally as well.
With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.
Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)
The easiest thing to do (takes 5 minutes) is to use LMStudio and then search for the model inside LMStudios UI.
It will show you on the model selection IF a model fits and if you use a "Qwen 3.8-35B-A3B" type of model, it should just work out of the box. A3B stands for Active which means you need a lot less fast ram.
You will run into issues fast though: If you have big tasks for it to do, your pc/mac will get warm very fast and noisy and you will cosume relevant more energy.
My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).
I get around 40-60 tok/s depending on the quantization level I choose.
As you can see, the CPU is nothing special, don’t even need DDR5.
Yes it runs on all levels of hardware - with the latest Strata my 5090 and 64GB DDR5 gets 200-240 tok/sec decode, 6500 tok/sec prefill. Around $0.09/M tokens in power usage (have excess solar capacity).
Jealous, lol. But I did spend a grand total of $500 for my setup. The rest of the server was built purchased with the use case of running game servers.
AI harnesses like Claude will set it all up for you, you can just say "I want to run a localllm on this machine" and it will find the best one for your hardware and even help you fine tune it etc
You will end up with a sub-par result. The space for small models is more specialized and specific and you should optimize to the specific task and or tasks you want to do. It will get some local model working, but there are a lot of interesting small models in the 4-10B space to look at and play with for your specific tasks. If you expect a full agentic coding agent it's just not going to happen without a larger model like Qwen 35B and ... that is not that great by today's standards, even Muse Glimmer is better.
After arguing and back and forth with Claude (not straight away), it found a kv streaming fork of llama https://github.com/achyudh/llama.cpp and an interesting ascii only model Qwen3.8-27B-ASCII-Condensed-UD-IQ4_XS.gguf . It's running, i'll see how it goes.
Ahh. Yeah. Well that sparks a memory of why I quickly went another direction after looking at it back then when I was learning web dev lol. Tbf a lot of people were, probably still are, using PHP and JavaScript way back then too. It just made sense for me to do some MVC style separation long before I found out there was an acronym for it.
This style is often how I want to write my view layer. In fact, when I get my model layer, app logic layer, and view layer done well, it's usually how my view templates will look more or less, whether the remaining data anchors, transformation functions, and control flow are out-of-band script or in-band tags.
There has to be some code in the view but I tend to just like to do pass it in a 'print ready' format so there's nothing really going on in the view except the presentation part. An if statement is often an indicator I'm doing too much in the view. Not always ofc, sometimes it makes sense to see if the count is 1 or >1 so you can plural a word, stuff like that is OK in the view. These are just my personal heuristics of course
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
reply