Hacker Newsnew | past | comments | ask | show | jobs | submit | Powdering7082's commentslogin

> redundant information in the structure that could be used to store other meaning.

Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)

That being said the attention heads are a small portion of the total network size


Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.

But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).


I think that in the process of actually struggling to craft a game people end up just doing lots of playtesting and at least if someone produced something as polished (visually and technically) as these are it'd be somewhat fun at least for the person that made it. As in it used to be very unlikely to get something that is technically complete and visually polished, but not game-play polished.

The site is down, but I saw some of them before it started getting rate limited.

It's an Opus 5.5 game and not one shot, but I have really enjoyed porting Cataclysm Bright Nights to 3D & real time: https://cata3d.com/


Back up and running smoothly now, thanks for your patience!

Obviously it doesn't affect the electric vehicles but light trucks being excluded from CAFE def did a lot to acculturate americans to always buying a big car

https://en.wikipedia.org/wiki/SUV#1980s_to_1990s


This seems to have kicked off a bit of an arms race. Smaller cars are more dangerous to drive as other cars get bigger.


> I turned it on briefly to receive a confirmation code that I needed in order to maintain that phone plan itself, and once spiritually but not literally - when going on a remote and slightly dangerous hike, friends insisted that I bring along a few communication devices that they had for emergency comms and use them to check in once every few hours.

It is funny to have witnessed the conversion from the default state of people is being not connected digitally to being connected digitally and once people start living with the 24/h connection they start the perceive simply being out of cell network as carrying inherent risk.

I've noticed people reacting to going hiking beyond cell networks with a "but that's dangerous!" response. Being outside of cell networks was the natural state of being for the vast majority of human existence!


> I've noticed people reacting to going hiking beyond cell networks with a "but that's dangerous!" response. Being outside of cell networks was the natural state of being for the vast majority of human existence!

And it was dangerous.

Also: I grew up just before cellphones started to be a thing and I felt a lot of times as a kid the frustration of not being able to contact someone remotely.


-https://github.com/jepsen-io/particledb is currently 404 (probably still private?)-

EDIT: It's up now


Sorry about that--yes, it's still private. Hoping to have that resolved shortly.


Looks like it's up now. I'll put that link in the toptext as well. Thanks!


> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof.

Perhaps they meant formerly, probably just a speech to text problem


TLA+ doesnt prove the code. if they never read the proof, then how do we k ow that it proved the right things?


And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.


lol, reminds me of the common misreading "provably safe" / "probably safe"


I am seeing the following in the reasoning trace:

> However, the instructions say I must use available tools when answering questions.

On "What animal lays the biggest egg?" with no system prompt

I assume this is something baked into the model through lots of RL with tool use?


Which specific model? and are tools available? I would say that there is lots of RL with tool use, but that the reasoning traces are not always perfect. In general, models think about things that might be wrong, but will recover from this poor line of thinking.


This was using llama-serve on my local which by default gave it current time & unhelpfully "get_info" which is specific to the browser

The full request was

``` {"messages":[{"content":"What animal lays the biggest egg?","role":"user"}],"return_progress":true,"sse_ping_interval":1,"stream":true,"tools":[{"function":{"description":"Returns the current local date and time in ISO 8601 format, with the IANA time zone name","name":"get_datetime","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"},{"function":{"description":"Get runtime info (OS name), may call when user asks about local files or shell commands","name":"get_info","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"}],"reasoning_format":"auto","reasoning_control":true,"backend_sampling":false,"timings_per_token":true} ```

Disabling tools removes the bit about mentioning instructions.

Just an interesting detail I noticed.

Do y'all have any penalty/regularization w.r.t. reasoning traces or is it pure SFT -> RL?


interesting, I didn't see it on the playground or on my local llama-server.

SFT -> RL. In our fine-tuning notebooks, we typically say that rollouts that go past the max context length (too much reasoning) get 0 reward, so there is slight pressure for shorter reasoning, but I don't think we have strong regularization. I think the agreed-upon recipe is to first make the model smarter (longer reasoning), then teach it to reason faster/denser.


(I work at Liquid.) Probably, yes. Per the blog, the last 3.5T tokens of pre-training were a mid-training stage with a larger share of agentic traces and tool calls, and SFT covered tool calling and agentic traces too, because the text model is post-trained to live inside agent harnesses. With an empty system prompt it seems to assume it's in one. I'd try a short system prompt that says no tools are available, and if it still does it, a note on the Hugging Face model page would help the team.


Do you use API for google & anthropic or subscription? I was under the impression those two are apt to ban you for using their models in a non-approved way


What exactly would be the tragedy of the commons here?

Publishing a crap app on the app store doesn't make anyone else's app worse


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: