> redundant information in the structure that could be used to store other meaning.
Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)
That being said the attention heads are a small portion of the total network size
Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.
But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).
I think that in the process of actually struggling to craft a game people end up just doing lots of playtesting and at least if someone produced something as polished (visually and technically) as these are it'd be somewhat fun at least for the person that made it. As in it used to be very unlikely to get something that is technically complete and visually polished, but not game-play polished.
Obviously it doesn't affect the electric vehicles but light trucks being excluded from CAFE def did a lot to acculturate americans to always buying a big car
> I turned it on briefly to receive a confirmation code that I needed in order to maintain that phone plan itself, and once spiritually but not literally - when going on a remote and slightly dangerous hike, friends insisted that I bring along a few communication devices that they had for emergency comms and use them to check in once every few hours.
It is funny to have witnessed the conversion from the default state of people is being not connected digitally to being connected digitally and once people start living with the 24/h connection they start the perceive simply being out of cell network as carrying inherent risk.
I've noticed people reacting to going hiking beyond cell networks with a "but that's dangerous!" response. Being outside of cell networks was the natural state of being for the vast majority of human existence!
> I've noticed people reacting to going hiking beyond cell networks with a "but that's dangerous!" response. Being outside of cell networks was the natural state of being for the vast majority of human existence!
And it was dangerous.
Also: I grew up just before cellphones started to be a thing and I felt a lot of times as a kid the frustration of not being able to contact someone remotely.
And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.
Which specific model? and are tools available? I would say that there is lots of RL with tool use, but that the reasoning traces are not always perfect. In general, models think about things that might be wrong, but will recover from this poor line of thinking.
This was using llama-serve on my local which by default gave it current time & unhelpfully "get_info" which is specific to the browser
The full request was
```
{"messages":[{"content":"What animal lays the biggest egg?","role":"user"}],"return_progress":true,"sse_ping_interval":1,"stream":true,"tools":[{"function":{"description":"Returns the current local date and time in ISO 8601 format, with the IANA time zone name","name":"get_datetime","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"},{"function":{"description":"Get runtime info (OS name), may call when user asks about local files or shell commands","name":"get_info","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"}],"reasoning_format":"auto","reasoning_control":true,"backend_sampling":false,"timings_per_token":true}
```
Disabling tools removes the bit about mentioning instructions.
Just an interesting detail I noticed.
Do y'all have any penalty/regularization w.r.t. reasoning traces or is it pure SFT -> RL?
interesting, I didn't see it on the playground or on my local llama-server.
SFT -> RL. In our fine-tuning notebooks, we typically say that rollouts that go past the max context length (too much reasoning) get 0 reward, so there is slight pressure for shorter reasoning, but I don't think we have strong regularization. I think the agreed-upon recipe is to first make the model smarter (longer reasoning), then teach it to reason faster/denser.
(I work at Liquid.) Probably, yes. Per the blog, the last 3.5T tokens of pre-training were a mid-training stage with a larger share of agentic traces and tool calls, and SFT covered tool calling and agentic traces too, because the text model is post-trained to live inside agent harnesses. With an empty system prompt it seems to assume it's in one. I'd try a short system prompt that says no tools are available, and if it still does it, a note on the Hugging Face model page would help the team.
Do you use API for google & anthropic or subscription? I was under the impression those two are apt to ban you for using their models in a non-approved way
Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)
That being said the attention heads are a small portion of the total network size
reply