Hacker Newsnew | past | comments | ask | show | jobs | submit | randbyte's commentslogin

So just go from bad to worse? I thought the lesson should be trust neither and pick an open weight provider from the dozens?

Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.

Were you “riding the line of potentially ban-able usage” with OAI though?

Absolutely not. No reverse engineering, no license infringement, nothing that could remotely resemble “inciting violence”. We are working exclusively on an accounting workflow based SaaS for law firms.

Yeah then that’s the risk. The compliance flagging is completely opaque and error-prone. And there is no recourse. You are now just betting Anthropic won’t flag you as non-compliant for reason only they know.

What happens if Anthropic bans you? Where will you go? Perplexity? Do you have a contingency plan in place?

Performative and just try to shed liability.

You conveniently left out the “done legally” part

It’s the wrong way by having a neutered market leader instead of strengthened competitors.

Guys going to start his own company. Surely?

He may not feel like putting out the effort. Hired by AMD could be interesting.

It looks rather generic. And to be honest I was able to use an agent to hand roll cluster serving stack using vllm, completed with spec decode, kernel cache reuse, and fixing or forward porting engine bugs.

You can definitely use an agent to help Day 0 deployment but we found this information useful while operating a platform with SLAs for real users (see this series - https://www.upbound.io/blog/drinking-our-own-champagne-part-...).

Feel free to help contribute to the guide also!


It’s a 32 node gb200 cluster across 3 az (with one sitting in house), running 3 different open weight models. The remote az are subjected to preempt all the time too. So it’s much more than a day 0.

My point is that no one should actually read a guide like this and rollout the stack manually. A bootstrap agent can do all these by reading the engine-model cookbooks and combine with distributed system common practices (which the models know quite well).

The “cutting edge” issues such as engine bugs for new model releases, topology and spec decode conflicts, are what the models don’t necessarily directly know and needs human collaborations. They are very situational and is not covered anywhere (and shouldn’t be).


So you think learning about inference engines is a waste of time as a career move?

I believe inference engineering is already an important emerging role which will continue to grow.

In fact today, CNCF at Kubecon announced a K8s inference engineering certification - https://cloudnativenow.com/kubecon-cloudnativecon-north-amer...

Our field guide helps folks get started in this area and the hope is that will be a valuable, practical resource for folks in their careers and work, as well as a valuable resource for folks using agents to help stand up their inference.


Except it’s not? Local bundled static electron and node is neither performant or safe…

BunElectron feels very performant. But I guess you don’t really want to know.

That’s not what OP is saying at all. Pi is another coding agent.

I honestly didn’t look into pi and inferred editor not agentic tooling. I’m one of the Claude Code focused people and almost never chase shiny things. The margins are never enough to get me to switch.

Hard disagree. I own my gb200 cluster and have unlimited supply of open weight tokens. Whatever is missing from the open weights, they can make it up from speed and multi agent swarm.

Yeah: quantity is a quality all of its own. DS v4.1 Flash running at ~300tk/s makes its verbosity less of a downside, and the efficiencies the Chinese open model space has aimed for makes high concurrency easy for swarms/sub agent sets.

On the other side, I am still shocked at how good Qwen 3.8 Flash Next is on my GB10 Spark-like, even if it is only 45 tk/s decode!


2 gb300 node actually. Or 8 gb300 cards since 1 node has 4.


NVIDIA HGX B300 seems to have 8 Blackwells, but I have a hard time getting concrete information from the NVIDIA website on what it actually is, and how you operate it, physically. The picture has this intermediate-product vibe that it should be part of a larger system, like the GB200/300 ones, and I feel like a little boy looking through the window of some shop with incomprehensible things. But I guess the whole point probably is that if you want to have it you put out a tender and someone else will worry about the design and integration and you just pay the bill and point to some empty space in your datacenter cage with large red 5-pole power bundles eagerly waiting.

If you look at the open weights official checkpoints they are either fp8 or even 4 (kimi). None is using bf16 for inference.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: