There’s a lot to discuss here, but I think the point most likely to hit home is one based in a) expertise and b) the nature of frontier labs. Specifically:
Would you know what a system that was capable of recursive self improvement would look like 2 months before it happens? Even 1 day? We don’t even know if it’s possible or what natural plateaus might exist for a DL-based system that rapidly evolves in su h a manner, so it feels hard to justify the other end, too (that it’s definitely not possible anytime soon). If anyone could do this, wouldn’t it be the scientists who specialize in this field? Cause they’re raising alarms in mass.
Assuming you make it past that first question with a “yes” and a “I’m close enough”: the huge amount of information assymetry and proven tendenency to hide safety concerns from the big labs is enough all on its own, IMHO.
Who knows what their most experimental, risky project is at any given time? Not us :(
Would you know what a system that was capable of recursive self improvement would look like 2 months before it happens? Even 1 day? We don’t even know if it’s possible or what natural plateaus might exist for a DL-based system that rapidly evolves in su h a manner, so it feels hard to justify the other end, too (that it’s definitely not possible anytime soon). If anyone could do this, wouldn’t it be the scientists who specialize in this field? Cause they’re raising alarms in mass.
Assuming you make it past that first question with a “yes” and a “I’m close enough”: the huge amount of information assymetry and proven tendenency to hide safety concerns from the big labs is enough all on its own, IMHO.
Who knows what their most experimental, risky project is at any given time? Not us :(