The systems need Lean, training material, benchmarks, data labeling, experts, publicity and more. OpenAI gave out free researcher accounts to collect more training material.
All of which was provided and encouraged by a small group like Tao, Gowers, Tsimerman, Avigad, Schmitt etc. It isn't just "an algorithm".
You don't think OpenAI has any interest in having agents identify new interesting research terrain? They just want to solve existing important millennium problems and then they're done?
Yes? I expect their motivation for devoting resources to solve these problems will dramatically drop as soon as they IPO. Maths is just a convenient vanity metric for OpenAI to demonstrate progress, but long term progress in mathematics has no commercial value to them.
Of course, general improvements may be sufficient to continue to drive math improvements even without specific attention - we’ll see.
> I expect their motivation for devoting resources to solve these problems will dramatically drop as soon as they IPO. Maths is just a convenient vanity metric for OpenAI to demonstrate progress, but long term progress in mathematics has no commercial value to them.
And why would IPO remove incentive that's present pre-IPO? Somehow math progress will influence price at IPO but won't influence stock price post-IPO? Or will they just stop caring about stock price once they are public. None of these options makes any sense.
No, post-IPO they will have to (at some point) demonstrate real commercial value. This just gets them to a stage to cash out insiders and (less importantly) get some additional capital to extend their runway.
That's the playbook, right - keep interest/investment flowing until you can figure out the actual commercial value/model. Until then, do whatever you need to keep the hype as high as possible and maximise your runway.
Not a chance. They'll go for room temperature superconductors and other stuff that is currently in the quite possible but not yet proven and patent the hell out of it and charge rent to the rest of humanity.
I think OpenAI's limitations are quickly verifiable results. 10,000 agents running in parallel aren't exactly talking to each other. They have a goal and one eventually hits the goal which can be automatically confirmed. This is why math is currently attacked, lean is easy to confirm A = B.
They aren't gonna create room temperature superconductors because they'll need a lab that can quickly, and verifiably, test superconductors at 10,000 per minute and that's just a physical limitation no matter how many datacenters you throw at it.
AI systems will probably be able to, if not direct labs themselves, cut down a lot of the search space to only the most promising candidates. That’s basically what Isomorphic Labs is doing for candidate drug research. It doesn’t seem crazy that the same could be true for material sciences.
Simulation to the point that the accuracy is large enough to extrapolate to real materials. You can do a lot of stuff when you have access to a million GPUs that would otherwise be much harder.
A lot of materials science is already done through simulation, same with a large chunk of chemistry and biology. That fraction will only increase over time.
I am aware of advances, but I think a large chunk is a stretch in various ways. We'll just see what happens - simulating even a glass of water fully is tough.
That depends on the granularity of your simulation. For most - not all - purposes a glass of water can be simulated coarsely or by simulating a far smaller amount with greater fidelity and then to extrapolate.
For useful materials science the numbers are still large but far lower than the numbers involved in 'a glass of water'.
I really depends on what is needed. If you know that something quite "coarse" is enough to create/uncover room temperature superconductors with high critical current - fair enough. Again, we'll see what happens.
With the kind of compute they have at their disposal 'quite coarse' may actually be 'impressively detailed' and that is why I think they have an edge on this kind of stuff. It's like being able to borrow a trillion and then to go play research lab, it would be highly surprising if they weren't able to monetize such a head start.
Eh, I'm sorry, but math being a vanity metric is not at all as factual as a take as you present it as. Maths are just a convenient target to improve their models through RLVR, using verifiable endpoints (ie. lean proofs) to generate novel training data. The publications and vanity stuff is just a side effect of their primary motivation: making the model better.
> The main practical complaint is "what do I do now", especially from PhD students who just lost their thesis topic. They sound like the Uber drivers fighting Waymo.
I wish mathematicians would just focus on this. This really is the problem, at the end of the day.
Having models get better at writing elegant proofs and explaining their results will not solve this problem - it will only make it worse, if anything.
We are all sympathetic to this. I, personally, am much less sympathetic to the absurd notion that it is "bad" that this new technology can solve important open mathematical problems at all, that we should stop trying to solve them, that things like the Mathathon are dangerous and needs to be shut down, that ugly proofs are the real problem here, so much so that you wish the open question weren't resolved at all, and so on. So silly.
But being pissed off that people are going to be out of work is something I agree 100% on. Hopefully, much like the internet, there's a way to make the whole thing a net benefit for society.
Yeah I view it as AI being objectively good from a technological perspective - we get new medications, new technology is achievable faster overall.
But really bad from a social perspective - as people lose their jobs they also lose their tiny piece of power in the society that was the labor someone else needed. Now that power has concentrated to the frontier model owners, and everyone in the future will depend on them.
And on a personal level, sacrifices people have made to become good at their craft have been made worthless, as any rich PoS can now prompt their way to "good" art and call their end result as good as something made with skill that needed years of training. That I really hate.
> prompt their way to "good" art and call their end result as good as something made with skill that needed years of training.
Which is only possible because:
a) The models have been trained on work of millions of people,
b) They've done it for free, In other words, they've stolen entire people's lives. Millions of lives. They literally suck people dry and tell them to go look for another job.
I wonder if there are proofs regarding what neural networks like the transformer can solve. Because fundamentally, Astra is just a bunch of matrix multiplications and a few well defined non-linear functions.
Anyways, most of the times, SAT can be solved quickly, but not always, so getting the answer from Astra in polynomial time doesn't prove much unless you can also prove that it always does, no matter the input.
Iirc, there's a big difference between the computational complexity of what a transformer can solve and what it can solve in practice; what it can solve in practice tends to come from things like singular learning theory (which describes the forms of generalizations that a given ML algorithm can make), but I think (iirc) that transformers can compute a pretty broad class of O(N^2) functions?
It depends on what you say is O(1). A theoretical Turing machine has an infinite tape that can be accessed in O(1), but in real life, it would require a machine of an infinite size.
Same idea for that transformer, it needs an infinite context, and it is an infinitely big machine, but with that level of abstraction, it processes a token in O(1).
In practice, there is no such thing as O(1) on arbitrarily sized data, there is no such thing as infinity either, it is just an abstraction.
It’s not the same for transformers, especially not with the theoretical infinite context. It simply doesn’t process tokens in O(1) in your given setup.
In practice, with frontier LLMs, it’s of course more complicated, and also they don’t reveal how they do lookups, but still the theoretical limit applies regardless.
Do you mind elaborating or perhaps providing some reference as to why you think transformers should be similar to a Turing machine on this specific criteria?
The context can change each time. To implement a UTM only requires 2 states and 3 symbols or the reverse.
So if you use the context not as a tape in itself, but as an io channel to read/write an infinite tape, the context for any given step can remain fixed in size, and so the step itself can be O(1).
Any basic lookup table is turing complete if you bolt an external infinite tape and read/write head onto it. But then that’s moving the goal post since the context wouldn’t be the “limitless tape” and it’s not a transformer “in the normal way” because turing completeness would be coming from external io and not from auto regressive self attention. Different topic.
The point is that the discussion is essentially entirely meaningless, as it's pretty much hard to implement any simple construct that can make deterministic decisions based on a data stream and wrap it in a loop without ending up with a UTM.
The consequence of that is that the original question GuB-42 raised is pretty pointless too. An "inert" transformer effectively has limitations, but as typically used it trivially becomes Turing complete, and so the class of problems it can solve is the exact same class of problems almost any of our other computational devices can solve.
And if you want to point out that this externalises all of what actually matters if we are talking about e.g. intelligence, then yes, it does. But by extension you can trade complexity of the tape for complexity of the machine, and so you can internalise any part of the problem that can be solved with a finite section of tape as a single state transition (trivial proof: For a given machine M with finite state, you can make your machine N execute those steps as a single state transition by having N run M on a state transition.)
To extend that to where I assume they'd want to go in the first place: A transformer wrapped in a loop with access to "just" a "tape head" can "only" be limited from being theoretically able to get all the way to human level intelligence if human brains depend on computations that exceed the Turing computable. But that is no different from any other Turing complete device.
Of course that says nothing of whether it's a practical way of getting there.
Of course it is. You’re conflating a turing machine with a transformer. Transformers don’t have a local read/write head. The [self] attention is over the whole context, which makes each step O(N) at best (with caching) and O(N^2) w/o.
of course it has a read write head. In fact it has N read / write heads equal to the size of the context since each latent state is updated with the results of the cross attention mechanism at each step.
Alternatively, you can view the '<unknown>' token that is dynamically driven to the embedding of the output token to be the write head. Either way, the simple truth is that you can train a transformer system (i.e., an iteratively applied transformer) to simulate an arbitrary turing machine.
There’s no such thing as a read/write head in transformers. You can use it as a metaphor but that’s a leaky metaphor, hence all these misunderstandings.
> In fact it has N read / write heads equal to the size of the context
And what is the asymptotic notation when computation scales linearly with the size of context?
> Either way, the simple truth is that you can train a transformer system
Which still doesn’t mean they have a read/write head or have O(1) complexity for token computation with “infinite context”.
> There’s no such thing as a read/write head in transformers. You can use it as a metaphor but that’s a leaky metaphor, hence all these misunderstandings.
The input to a transformer is a list of embeddings. The output is a list of embeddings of the same length. At the end of the day, you can represent the entirety of the transformer's matrix multiplies and such as a DFA (IEEE 754 float multiplication and addition is a simple DFA, and you're just scaling this up that many times). Thus, you take a list of embeddings (i.e., the tape) and output another, modified, list of embeddings. You have successfully read (ingested) and written (the transformer output) N positions on a tape.
Yes and no. These things have always existed and were typically known but not broadly exploited. A huge part of what nation state hackers do is exploit these things for intelligence use.
So I would agree that this is bringing more of this to light, but the idea that unknown exploitable bugs can get weaponized being a new thing is naive.
If anything putting all this out in the open makes things more secure, not less, in the long run as it’s harder for hackers to know and exploit them before they’re known as they don’t survive undetected for long. So a bit painful in the short term but we get more secure infra in the long term.
Still not a reason to give Sam regulatory capture as an out to saving OpenAI.
Roughly speaking, O(n log n) means "grows no faster than k(n log n) for some k", whereas o(n log n) means grows strictly more slowly* than k(n log n) for all* choices of k.
"The psych had told me nothing of the nightmare that dependence on those meds can create"
I hear this all the time and never get it. Didn't these people go to school? People talk about this every day on the internet. It's right there on Wikipedia, even.
They're dealing with neurotic (not being judgment, these are a powerful class of drugs) people. Like lines of code, (saying) less is more. "oh by the way, some people get how's
hooked on them and it really sucks to get off them". then the patient doesn't take them and get the relief from the problem they saw the psychiatrist for. Any responsible psychiatrist is well aware of the problem and going to get their patient past their current crisis first, and then not give them a script for 18 months for the patient to build an addiction. Unfortunately, not all psychiatrists will do that.
Yeah, that’s a classic dilemma doctors have to face. Even my gastroenterologist stopped giving me details as soon as she convinced herself my issues were due to anxiety (they were not, but that’s another story).
> History judges researchers who work with the bad guys ... To them I say: learn from history and get out now. You’ll still be ahead of the curve. The current backlash against AI may seem huge, but it’s nothing compared to what I think the backlash will be in coming years.
> I chose 7 years because I see certain parallels between the situation with researchers and Big AI today and the relationship between researchers and Epstein in 2019.
Look, I absolutely love Vi Hart's stuff. And I mean love love, and I have for almost 20 years.
But do people really believe this? That associating with AI companies, using their product, etc, is similar to taking money from a child sex trafficker?
I thought the original Mathathon sounded like a great idea. They shut that down, and I think the revised one seems like a half decent idea. But if I get involved in it, in 7 years it'll be the same as if I had been involved with Jeffrey Epstein?
I feel like the discourse on this is starting to get way out of control. I say that with no less love for Vi Hart's ideas than I have always had. But surely this technology can be used for some good?
(a) AI will replace all or nearly all human intellectual labour
(b) AI will achieve superintelligence and become a (hopefully benevolent) digital god
(c) AI will be a superweapon that defines the next era of warfare
(d) AI will render art and creativity obsolete
These are real arguments I have heard. I'm sure you have too. If any of them are true, even partly, would you really be shocked to see the people of the future looking back at us as a bunch of Chamberlains?
And would they judge us less harshly just because the people who wanted to end the economy, summon Cthulhu, build a nuke and kill art simply failed to achieve their goals? Is the goal itself not odious enough to condemn?
None of it is ridiculous in principle, but where do you draw the line? Is Google an AI company whose products I shouldn't use? What about Microsoft, where Vi worked for many years? Or is it specifically OpenAI and Anthropic? What about open source companies?
People are still arguing about capitalism vs communism, even though the laissez faire capitalism that inspired literal communism ended with the New Deal and literal communism fell some time around the Berlin Wall (give or take when China became 'state capitalism' in practice if not name).
One thing is, I am confident, going to be true regardless of AI: for as long as humans are still around, we'll be arguing about the economy and what 'good' even means in that context.
Less of a quibble:
> (b) AI will achieve superintelligence and become a (hopefully benevolent) digital god
While supremely hubristic, is that really odious given the 'hopefully benevolent'?
Like, I don't believe in any god, but is this any more odious than someone simply praying for success to a god they do believe in?
Some of the smartest people I know are also some of the people that use AI the most.
They use their time and energy on solving hard problems. Either in the abstract (research) or more practical problems (entrepreneurship).
I completely fail to see how anyone would want to equal researchers and entrepreneurs to criminals.
Plus the dozens of existing cyberattacks initiated by OpenAI and Anthropic that were simply shrugged away by their leadership.
[1]: And there's a related Tweet I can't find right now, from an AI startup founder who agrees and encourages people to start prepping supplies.
Once shit really hits the fan, I expect a good majority of people will come to view AI as a toxic force that sows discord and saps the life out of everything they once enjoyed. They'll witness endless cyber warfare, all-enveloping disinformation campaigns, devaluation of art and culture, mass unemployment and unbelievable oligarchic consolidation, spam and enshittification at unprecedented levels, incredibly complex phishing and ransom schemes, every last bit of commercial human interaction turned over to bots, complete calorie-free information overload in exchange for... what, exactly? An improved Google search that confidently tells you the wrong thing? A digital avatar that makes trip planning marginally easier?
> This is our view as well. Relatedly, average people will want to increase their level of personal resilience and preparedness as their resources allow. We will be publishing about this soon... This is why I bought a bugout property towards the end of last year.
> But do people really believe this? That associating with AI companies, using their product, etc, is similar to taking money from a child sex trafficker?
I'm not saying he did it (how would I tell?), but Altman has absolutely been accused by his own sister.
And while Musk doesn't seem to have actually gone to that island, he did seem interested.
And one of the files did show Zuckerberg and Musk dining with Epstein.
And many of the big tech leadership did openly support Trump, despite the long list of things that now includes suspicious gaps in which of the Epstein files actually got published. (Even if theres a legit reason behind what remains hidden, promising to publish then failing is at best still a huge own-goal because "they have something to hide" is necessarily true and always sounds like it means "that they themselves did").
So, yeah, I can see it being that level of bad for at least several of the big names.
(There's also a lot of distaste for Anthropic; I'm not aware of anything of this category, but even without that it is easy to tar everyone with the same brush).
This is a very loaded blog post. A unit-cost RAM is a fairly standard model of computation when working with algorithms. When we say that comparison sorting of strings takes O(n log n) time, for instance, we are also implicitly assuming a unit cost for comparing two strings. In real life, comparing two strings scales linearly with string length. Maybe that's silly, but it's par for the course. The original FFT is also assuming unit cost multiplication when we say it is O(n log n) time - we mean O(n log n) multiplies.
Similarly, yes, it's a galactic algorithm, but these things are still viewed as meaningful. The fact that it is even possible at all with a unit cost RAM asymptotically is an incredible result that nobody expected.
The string comparison model is useful in the regime where string lengths are much shorter than list lengths and can be bounded by a constant. It would be a horribly broken (i.e. unrepresentative) model if you were constructing intermediate strings whose lengths could grow arbitrarily and become much longer than list lengths.
One can of course make a puzzle out of anything, but it behooves asking what situation is being modelled by that problem.
And yes, this criticism might apply more broadly to TCS rather than just this specific artifact from OpenAI. It's reward hacking regardless of whether humans do it or LLMs.
reply