Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.
Absolutely not. No reverse engineering, no license infringement, nothing that could remotely resemble “inciting violence”. We are working exclusively on an accounting workflow based SaaS for law firms.
Yeah then that’s the risk. The compliance flagging is completely opaque and error-prone. And there is no recourse. You are now just betting Anthropic won’t flag you as non-compliant for reason only they know.
It looks rather generic. And to be honest I was able to use an agent to hand roll cluster serving stack using vllm, completed with spec decode, kernel cache reuse, and fixing or forward porting engine bugs.
It’s a 32 node gb200 cluster across 3 az (with one sitting in house), running 3 different open weight models. The remote az are subjected to preempt all the time too. So it’s much more than a day 0.
My point is that no one should actually read a guide like this and rollout the stack manually. A bootstrap agent can do all these by reading the engine-model cookbooks and combine with distributed system common practices (which the models know quite well).
The “cutting edge” issues such as engine bugs for new model releases, topology and spec decode conflicts, are what the models don’t necessarily directly know and needs human collaborations. They are very situational and is not covered anywhere (and shouldn’t be).
Our field guide helps folks get started in this area and the hope is that will be a valuable, practical resource for folks in their careers and work, as well as a valuable resource for folks using agents to help stand up their inference.
I honestly didn’t look into pi and inferred editor not agentic tooling. I’m one of the Claude Code focused people and almost never chase shiny things. The margins are never enough to get me to switch.
Hard disagree. I own my gb200 cluster and have unlimited supply of open weight tokens. Whatever is missing from the open weights, they can make it up from speed and multi agent swarm.
Yeah: quantity is a quality all of its own. DS v4.1 Flash running at ~300tk/s makes its verbosity less of a downside, and the efficiencies the Chinese open model space has aimed for makes high concurrency easy for swarms/sub agent sets.
On the other side, I am still shocked at how good Qwen 3.8 Flash Next is on my GB10 Spark-like, even if it is only 45 tk/s decode!
NVIDIA HGX B300 seems to have 8 Blackwells, but I have a hard time getting concrete information from the NVIDIA website on what it actually is, and how you operate it, physically. The picture has this intermediate-product vibe that it should be part of a larger system, like the GB200/300 ones, and I feel like a little boy looking through the window of some shop with incomprehensible things. But I guess the whole point probably is that if you want to have it you put out a tender and someone else will worry about the design and integration and you just pay the bill and point to some empty space in your datacenter cage with large red 5-pole power bundles eagerly waiting.
reply