Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)
- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
Cautionary note (and a serious one, not to be snarky): Being well off financially is not a protection against gambling addiction (which can also take the form of day trading, for example); which is a pretty efficient way to ruin ones financial wealth, amongst other things.
As with many other psychological phenomenona, finances and education only protect to a certain degree or not at all, depending on what one is talking about in detail.
This is not a comment about correlation or causation of wealth with mental health, but rather a warning to keep in the back of ones head; success and a sharp mind don't make one immune against mental health issues in general and gambling addiction in particular.
To me, the difference is a form of classism. Addiction to drugs is for the lower class. Addiction to gambling from making money from investing is something only upper class people can have.
I feel like this is just straight from the Kalshi marketing department. The wealthy make money from speculation by taking calculated risks and offsetting those risks with other investments. They're not betting everything on black at the roulette table or bettin on the outcome of football games.
Always nice seeing others arriving at similar conclusions! Those two jump out to me:
- "Most compilers hard-code a cost model. Vx reads one. A machine file describes the memory hierarchy and interconnect of a real part, and the compiler admits or rejects placements against it." I also designed it so that the target system is described by a single file that is picked up by the static analysis - cool!
- "Where data lives is part of its type" also doing that - when I got my project to an mvp state; I need to check out how vxlang does what it does: do they use a literal typesystem, or also compile checks outside of that? If its a type system, is it a dependent type system, or another flavor?
Yet another motivation to finally get my side project starting (forth-adjacent, many similar claims with regards to compile-time checks as vxlang; however, I still have to achieve that, while they already seem to have at least those and more in place; kudos to the vxlang team!)
The main problem I see with RC which has prevented me from applying is there’s no stipend or housing. Most people can’t afford a summer in NYC without pay. So I’m not sure who the target audience really is, I guess wealthy SV engineers.
Not really something you can control, but RC always feels "unauthentic" whenever people write about it.
It is often portrayed like an old-school hackerspace or makerspace but it's definitely not. It has an explicit corporate background, and it always feels like that is somehow hidden.
In these comments there is a comment that claims that RC is not for business builders and networking, then immediately gets disproven by a quote from its very website. This shows that you have an actual image/brand problem.
It is similar to how the term "hackathon" evolved. In the old times it was people coming together who worked on shared goals (like a FLOSS project) but only communicated via mailing lists and probably would never met in person otherwise. A hackathon was often an emotional thing, involving poor people travelling to the other side of the planet to meet their online friends to drink beer and finally have a whiteboard to communicate some complex ideas. Then corporations came and organized soulless "hackathons" and turned it into an idea farm and talent recruitment. The term "hackathon" is pretty much burnt completely nowadays.
So anything that feels like it gets subtly used by corporatism just naturally gives techies the ick. It is not the first time this bait-and-switch happens, so people are fair to assume intent.
I say you cannot control it, as obviously the public image comes from those blog posts of RC "Alumni" (Another great example: that term alone gives me the creeps). And those people have the right to feel excited about a nice experience that it probably was for them.
But Recurse Center should probably be proud of the corporate background and actively counteract the impression of being a hobbyist makerspace that just has better inventory and manpower.
As a software engineer, I could not care less about how a bug was found or by whom, as long as I can quickly verify it is correct, I always appreciate being able to improve my work. I don't see why it should be different for mathematicians (the ones I knew would think similarly, I would assume.)
If I may offer an analogy, what I tried to do is essentially reporting a bug after having a failing unit test that exercises the public API, without looking into the black box of internals.
> Please do not ask the LLM to break the universe :)
Eh, every school db should be able to service a student called "Robert'); DROP TABLE Students;--", and every public facing LLM should still work even if asked to calculate the last digit of pi - that fulfills two noble purposes: 1) is fun and teaches whoever is responsible for the service some valuable lessons in craftsmanship or 2) if the service is built robustly (i.e. simple token limit and time limit per user), that's an easy way to gain street cred
> If this turns out to be true I think my take above is not correct, in the short term people will have to stop sharing updates because otherwise openai will dishonestly race to finish their work.
Which I don't see a reason for Anthropic and "Open"AI not to, given their not so stellar track record with IP of individuals/entities-that-are-not-rich-enough ;)
"But I realized after a while that talking to people casually about Fermat was impossible, because it just generates too much interest, and you can't really focus yourself for years unless you have this kind of undivided concentration, which too many spectators would have destroyed."
But yes; him reaping the benefits of himself having the idea first was part of it too; as far as I am aware.
-----
Which is still something completely different than some anonymous organisation keeping mathematical research secret because it is better for hype reasons. One is competition between individuals or groups within a field; the other is boring and sometimes borderline nihilistic generating of mathematical knowledge as an marketing asset.
It is a strong argument in this case though, because Terence Taos expertise is directly linked to his ability to not misstate the history of mathematics.
Also note how the quote by Tao is in all likelyhood not meant as an absolute; rather than a statement of a trend - a handfull of counterexamples do I no way change anything about the truth value of Tao's quote.
On the other heand; consider how absurd it would be if "... in the direction of no longer sharing any promising research directions with the broader community, which would reverse centuries of traditions of open science ..." would indeed be a misstatement; which would imply that far more promising research directions were not shared with the broader community (i.e.: published). I wonder what different reading of that counterfactual there could be other than secret societies that kept their discoveries and research directions to themselves - which we just learned about (since we would otherwise not be refering to the secret societies and their supposed promising research directions).
All pretty straightforward, I would say - both that "misstatement" is hopefully based an overly strict reading of Tao's quote, and that mentioning Tao's background as one of the fields leading practitioners is relevant as well. Again; to make sure: A few counterexamples achieves nothing here. It would need to reach a certain threshold of such counterexamples before we will have to write the history of mathematics; and before Tao actually made a misstatement here.
I mean mathematics has enough history that something can both have been false for centuries of mathematics research and true for centuries. Sometimes in different places simultaneously.
Terrence Tao can do his job perfectly well without being aware of any mathematical history, though I consider it unlikely that he is. I'm not seeing the direct link you're talking about, in fact history is frequently left out of mathematical teaching even when the history would in fact help in the understanding of some concepts.
> in fact history is frequently left out of mathematical teaching even when the history would in fact help in the understanding of some concepts.
That is well-known I assumed and continue to assume.
> I'm not seeing the direct link you're talking about
You are stating that link yourself; indirectly: "though I consider it unlikely that he is [being unaware of any mathematical history]". Why is it unlikely, precisely?
- Maybe because it is unlikely that he recieved the mathematical teaching that frequently does not contain history of mathematics (wild! I wonder which university you have in mind in particular) that you seem to be refering to?
- Or maybe because he is quite the opposite of a person that never ventures outside of their own area; being blind for other fields, or ones own history; as evidence by being famously collaborative across different fields, having a popular blog where he writes about non-mathematical topics too and last; him being one of the main proponents of foundational topics such as formalization of mathematics; or the use of LLMs for mathematical research.
Does all that really make it more likely to you that Tao is not aware of the existence of counterexamples like those the commenter above mentioned - more likely than the commenter simply having missed a nuance or taking something out of context?
If so; I would be genuinely curious why - people work differently, and I am always happy to learn, or close gaps in my own understanding.
Any one of the arguments you make here is already stronger than the original one. The problems with appeals to authority isn't that they are always false, it's that they are not a good argument.
And you say his "expertise is directly linked to his ability to not misstate the history of mathematics". And frankly, I disagree. If he happened to be misguided or even outright wrong about some parts of mathematical history I wouldn't think any less of him, nor do I think it matters much for the work he's actually paid to do. At worst it would result in an online discussion, which is arguably a good outcome not a bad outcome.
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
reply