You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).
I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
This is an overreaction. Yes, with AI we can all write our own features and bug fixes for open source, but there’s still community benefit in having a canonical version somewhere, rather than everyone writing bespoke solutions for themselves.
> there’s still community benefit in having a canonical version somewhere, rather than everyone writing bespoke solutions for themselves.
There is sharply reduced benefit to the maintainer, however as they are now buried in slop bug reports and PRs, and a bunch of people with no context and a Claude subscription think they can do it better.
I expect a lot of open-source maintainers will walk away from that particular bargain in the near future
I'm not really advocating for any position here - just pointing out that open-source maintainer was a thankless role before LLMs, and it's only become worse since.
The story probably is being curated off the front page, but to be fair I understand the motivation. There aren’t many subjects that inspire as many high emotions as Israel. Everyone has strong options. The mods probably think it’s not worth the trouble to police the threads and they’re probably right. Nobody is going to change any minds in any of those threads anyway.
When the digital cameras first came out, they were pretty terrible. Low res, bad colors, small memory. By all rights the consumer should have taken a look at the difference in quality between film and digital and stayed with film. Certainly, if as was imagined in some circles, the average consumer was the kind of connoisseur that valued quality photographs.
The average consumer was not the kind of connoisseur that valued quality photographs. The average consumer wanted something cheap and simple that allowed them to take photos and skip bringing them in to the shop to have them developed. The average consumer wanted to have their photos on their computer without scanning them, and didn’t want to pay a Polaroid premium for instant shots.
Over time digital cameras got better and better until the technology was subsumed into phones. Few people are unhappy about how it went.
This is how these things go. An inferior good is made, competes with the incumbent on cost and convenience, and is improved over time to become the dominant kind of good. This is the definition of disruption. The software industry, after decades of doing the disrupting, is now being disrupted.
The issue they will hit is that without a direct economic incentive and paying customers, there's zero perceived benefit for fixing bugs if you don't even share in the joy of the work. Thus you end up with a buggy, free version of Adobe's incredible product suite that tries to ride the hype/feature treadmill in perpetuity.
There's plenty of users bringing their own Claude subscriptions to fix bugs so it's not all a cost on the initial creator. The motivation is to have free open source high quality programs.
There's already a huge market of OSS products that are much more stable and closer to real feature parity to the Adobe suite. How have all those users bringing their own Claude subscriptions worked out there?
On one hand we have established OSS operating perhaps on a principle of accepting AI generated contributions only after thorough review, if at all.
On the other side we have a vibecoded Photoshop clone with a new community where presumably there are very low hurdles to contribute more vibecoded implementation.
Let's see if this will produce a usable application.
Established projects do not want some drive by radical change set of PRs dropped on them. They mostly only accept bug fixes, and most of them do not accept AI generated code.
For what these projects are trying to achieve, starting a new project was the only real way. Can you imagine trying to convince the gimp maintainers to completely dump their current UI and adopt an AI generated Rust version replicating photoshop exactly? It would never be merged.
Your assumption is wrong. Consider, for instance, the LHC dataset, which reach O(EB). Just bringing the data into disks, loading into RAM in chunks, doing the data analysis and reduction, and doing the statistical analyses and fits, is a monumental task.
Consider now the work needed to operate the detectors, acquire the data, do calibrations and validations.
Notwithstanding designing, building and commissioning the detectors.
The interviewee is worried about the future of math research. He is not strictly worried about being replaced, instead he is worried that he will no longer be able to launder math-as-a-hobby through math-as-something-useful as is the case today. He lays out very clearly that grant proposals claim to have useful outcomes while the proposers know those claims are nonsense.
Business as usual in math, and frankly in all the other sciences, is to do research that furthers the researchers careers or personal interests and pretend that it’s somehow useful. This would be absolutely fine if it were privately funded, but it’s not, this is public money.
In every other endeavour, lying in order to get money is considered fraud.
We have collectively wasted a huge amount of taxpayer money and human time, entire careers, on things not likely to ever matter to anyone.
I look forward to science becoming automated so that we can have real progress instead of the current broken system.
People thought, back in the 17 century, that imaginary were useless (except as a trick for some calculations). Turns out the research into these numbers back then is amazingly useful today, 300 years later, in electronics and such.
Publicly funded maths research should continue, even if some taxpayers feel it's a waste of money.
Nobody determined the usefulness of the internal combustion engine or the aeroplane after the fact. They were goals specifically worked towards. For every useful discovery that comes from this hobbyist approach there are far more that aren’t, and those would have been discovered during a goal orientated research program.
However, with AI it actually may become so cheap that the scattergun random approach becomes more viable rather than less. It’s when human time and resources are scarce that you need to optimise. The hobbyist approach may therefore ironically continue, but without the hobbyists.
> usefulness can only be determined after the fact.
Thats great! So in other words it should be perfectly fine for AI to solve all of the supposedly useless math problems so that it might possibly be useful later. No need to worry about the mathematicians hobbies here.
> No need to worry about the mathematicians hobbies here.
of course not.
Hobbies can still be done even if AI gets it done more quickly. Just like today, where knives are made much faster/cheaper in presses and CNC machines, vs a blacksmith hammering. But still, there are hobby blacksmiths.
it's the same - what research getting funded should not be determined by the applied nature of the research.
In fact, the taxpayer should be funding fundamental research because it's so hard to justify profit from it - but that funding would benefit all in the long future.
So leaving the applied research that have commercial value be funded by private, commercial interests, would make more sense.
Actually yes, I think that AI will end up in a place where additional human effort, even at the highest level of suggesting what directions to look in, will become a rounding error compared to what the AI will achieve by itself.
So as measured by utility, I absolutely believe we don’t need humans doing science into the future. I’m sure people will continue doing it, but not for utility, for enjoyment - as a hobby. Probably we’ll all end up as dedicated hobbyists.
The whole point of public funding for science is that seemingly pointless research yields useful but hard to monetize discoveries. Otherwise VCs would be doing it.
Haha. That’s almost verbatim what the guy said. Here, let me grab that for you:
> things that I do and that my colleagues do, this kind of like curiosity-driven, you know, applied math, computational physics type research has always been justified by, I would argue, intentionally blurring the line between what I would call, you know, science as product versus science as process
> science as product is very kind of clear-cut. It’s, you know, things like, you know, cure cancer, solve nuclear fusion, generate, you know, clean energy.
> And then there’s science as process, which is kind of the curiosity-driven stuff about, you know, like, “I want to understand protein folding,” or, “I want to understand, you know, turbulence,” or, “I want to understand quantum gravity,” or something. And broadly speaking, we have tended to justify the latter by kind of laundering it through the former
And the examples he gives are actually the more defensible ones, he talks about a friend of his working on some abstract algebra under the false guise of cryptography research later.
And it’s not just him saying it, this is simply true. He should be lauded for admitting it publicly, this is the only way any progress is made. At least, it used to be. Now it’ll be AI instead.
Indeed. I wonder how much work it would be to make this antibiotic resistant. Then the Russians develop some bacteriophage that infects and kills the bacteria as their own cure.
Making it antibiotic resistant is not difficult and they already did that back when they were the USSR. Not only did they develop it, they produced large quantities of it to stockpile for a rainy day.
> Do we currently have the ability to do targeted methylation or de methylation at sites in cells?
No, and you probably wouldn’t want it if you could. You don’t know what mutations are lurking in the genes that have been switched off or the consequences of turning them back on.
I’ve thought about this a bit, and what resetting the methylation map really requires, in addition to actually resetting that map, is resetting your genome to a known good state.
That’s pretty radical, but to selectively switch methylation on and off with precision you’re already talking about science beyond anything we have, featuring custom engineered constellations of proteins that unwrap your chromatin from around histones and then walk along it changing methylation state, then pack everything back.
And doing it for each cell type, without applying the wrong map anywhere.
And knowing what all the cell types and their correct methylation maps actually are.
And assuming that methylation isn’t used anywhere, for example in the brain, in a way we don’t expect.
And methylation isn’t the only kind of epigenetic modification.
So, if we’ve solved all of those problems, rebuilding your DNA to a clean, mutation free state at the same time should be a breeze.
We really do need AI for this if any of it is going to be remotely feasible.
Cancer is the price we pay for complex multicellular life. It’s not a failsafe, it’s the failure of all failsafes to regulate cellular growth within an organism.
Evolution doesn't care about individual organisms, it's all about reproduction. Our lifespan is enough to raise a few children and pass genes on, so, from the evolutionary point of view, it's adequate
There's the grandmother hypothesis that holds we could just not have menopause and reproduce more, but we don't because there's greater evolutionary advantage for having the elderly provide support to their kin. Maybe this type of kin selection is bounded by consanguinity though, and that limits the advantage? Or maybe population pressure discourages family units from getting too large?
That’s the kind of evolutionary explanation that doesn’t fit natural selection at all in any time period
You would need to explain how populations who did those things were selected against
Instead of the more aligned reality that there was a random attractive or forceful menopausal mutant that popped out alot of babies or put alot of babies in others
speaking of attractive, what we see now is that older women are selected against and everything considered attractive is correlated to ability of carrying a pregnancy to term. which came first? impossible for me to say, would older women have many more pregnancies if the menopause never happened? the viable eggs would have still gone towards zero, unlike other mammals and beings
I understand, I'm equally frustrated when people including scientists say that something evolved for an environmental specialization, instead of the evolution being happenstance that may have incentivized more viable mutant offspring surviving until reproduction in that environment, but was not "for" anything.
and yes, so I understand that with human cancer, it tends to happen or advance after reproduction has already occurred so therefore those traits are not selected against. it's impossible to weed out with human mating strategies
I would just like to believe the cancer itself is an adaptation preventing something worse, but you're right, it doesn't work with the natural selection theory and so we just had a dead end to our survivability baked in, not so different than how the opposite phenomenon of senescence in cephalopods ends their lives
Everyone involved in this discussion understands that evolution doesn’t create things for a purpose, that it’s just the result of selection pressures. Everyone also uses the more active phrasing because it’s much more convenient. Unless you think you’re talking to someone who genuinely doesn’t understand the nature of the process, it’s not useful to be pedantic about it.
Evolution doesn't care at all, its just environment pressures and whatever organisms do within it. It can very well lead to some dead ends and sub-optimal long term choices.
There will come time when we will grok all this 100% and then can setup whatever we want (its just a bunch of molecules acting predictably), but we are very far from there and there is no point discussing this even for our grandchildren. We will have cyberpunk implants way sooner.
Evolution doesn't care about anything. Evolution is a descriptive force, not prescriptive. It boils down to the statement that things which copy themselves, we have lots of, which should seem almost like a tautology. It has no goal or purpose. It's an extremely random process and the apparent purposefulness is mostly just the result of that random walk having billions of years to run.
> wondered whether cancers in humans are a failsafe
I don't think so, but in the context of theories that are more-interesting than "stuff just breaks sometimes", I'd like to suggest the atavistic aspect.
Some cancers resemble a return towards unicellularity, as if "newer" multicellular rules have broken down. Not in any remotely-effective "our genes shall survive the dark times as goo" way, though.
> We really do need AI for this if any of it is going to be remotely feasible.
Would you give your life into the hands of AI skynet slop?
I understand your rationale, of course, but I for one don't want big fat mega-corporations dictate over our life here. Naturally, all of this hinges whether AI can even solve this problem. I assume it can, but I doubt it can do so initially - and the legal implications are still enormous. I really am not ready to turn my hands to claude's gene therapy slop.
I suspect that epigenetic aging is adaptive; a deliberately programmed sequence wherein genes more likely to contribute to age-related mortality are shut down progressively. This being in response to random genetic damage accumulating at a predictable rate.
What we think of as aging then becomes primarily the epigenetic response to the problem of DNA damage, much like the symptoms of a virus are mostly the result of the immune response rather than the virus’s direct effects.
There's a lot of indirect evidence for this from the fact that some naive attempts at stopping aging give you cancer, like turning telomerase on indiscriminately.
Almost everything in nature has multiple functions or causes, but one cause of aging is probably an evolutionary compromise between longevity and not getting cancer.
One option, I guess, would be turning the guard rails off and just getting incredibly good at treating cancer or inventing some extraneous cancer detection and killing mechanism.
It’s rarely such a simple tradeoff that only two things are being optimized. Modern humans live way longer than wolves for example despite being of similar body mass. However that’s fairly new in our evolutionary history as wild chimps for example cap our at ~63.
Cancer is extremely rare as a cause of death in wild animals. Again is more common but late enough that reproduction has produced multiple offspring.
Energy expenditure isn’t something we’re concerned with but drives a great deal of evolutionary optimization.
We all get cancer every day, and our body destroys it. When it finally fails to identify one, you get a tumor. If we’re doing something that massively increases cancer rates, we’d likely need a daily pill of some kind, or we’d be getting screened for cancer twice a year or something.
Can you state which genes these are that do that? Because I don't know of any.
Plus, many genes have pleiotropic effects. There is no specific aging gene.
On top of that, aging is a word that combines to many different factors. For
instance, progeria was called accelerated aging. Well, turns out you have a
mutation in lamin A. So, it is about cytoskeletal structure that is defect, rather than aging in itself. Of course the effects that this has, looks like an older person then, so it is related to aging. But one can not say it is "accelerated aging" as such. People having a normal lamin A allele still age nonetheless. So the whole term is problematic. And you can find many more such descriptions where xyz is about aging. Well, most of that it is about damage rather than aging. The Hayflick limit, though, is not about damage; that's just that cells fatigue for some reason when their telomeres shorten, but I have not yet read a sound explanations about why that is the case - after all there must be a mechanism in place.
I hope I’m just being melodramatic.
reply