Enterprise is not freaking out because DeepSeek 4.1 Flash does not actually occupy a spot on the Pareto frontier for non-coding enterprise workflows. We see this at my employer, focused on non-technical knowledge work. Luna 6 and now Haiku 5.5 are both very competitive if not better on all axes that we care about
Letting a model manage its own context is very bitter lesson-pilled
Biggest challenge is you will get a much lower cache hit rate if you frequently edit the agent's context/prefix, so this can not be implemented efficiently via e.g. the Anthropic API.
This ^ can be solved in principle but likely requires modifications to the transformer architecture and definitely to serving infrastructure
this is done on the model side so it is invisible to the caller. all of deepseeks kv cache compression stuff is basically the same thing. it doesn't really put the entire context into kv cache and does various inference side stuff to decide what to actually use as context.
Java spent decades making pointer-rich identity semantics cheaper, before increasingly recognizing that many things programmers call “objects” are actually values.
We may spend a decade making persistent token-level KV state cheaper, before recognizing that much of what we call “context” should actually be transient, compressed, reconstructed, or represented in an entirely different state space.
KV cache risks turning an optimization of today's representation of history into tomorrow's semantics of memory.
True facts. Cline does this with auto compact and it's so dumb. At the start of the session it reads the plan.md. during compaction, it forgets it. Same with its own output. It'll tell me the list of steps to take. By step 4 it needs to compact. So I prompt to do step 4. It spends 15 minutes searching the codebase and comes back that it couldn't find such a step. So dumb. So so dumb
Shocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
As someone who used priming heavily at Google, and its wide use in social networks to billions in revenue. I’m gonna go with the academics trying to take this down are BS.
Adam Curtis even suggests lightly that the takedown of priming in academia was paid for to bury its actual effectiveness (and liability by implication)
priming obviously works. it is inherent in the physical structure of the brain. nerve endings from various all over the body terminate in different physical locations in the brain, and then these are interlinked in an adaptive biological mesh network. nerve responses are triggered by cascades of correlated impulses across different nerve endings, so at a physical level a strongly correlated signal from any two major nerve groups will trigger a sympathetic response when one of them fires independently. the experiments that social scientists come up with may be flawed, but this is definitely how my brain works, so i don't need a study to prove it.
For some definitions of better, yes. Chinese is more token efficient for representing fixed text, for example, although this does not always lead to better performance on downstream tasks.
True. I suspect it's still hard to tell whether the bottleneck is the language itself, the tokenizer, or just the overwhelming amount of English training data.
As the author - this was adapted from a thread posted on X in March (linked in article). AI did the adaptation, I wrote the original article. It seems like it inserted grammatically correct hyphens, otherwise the copy is mine.
