Lots of anti-AI takes seem to fall prey to the moralistic fallacy (ie because something shouldn’t be, thinking it isn’t). I’m very skeptical of the claims AI companies make, their motives, and whether their impact will be anything positive. But I am pretty tired of takes like this that are so reflexively skeptical of anything AI that they carelessly dismiss claims about the dangers and real impacts of AI that are happening.
(And to be clear: I’m not talking about being skeptical of fantastic claims about things that have not happened- ie an AI takeover, runaway super-intelligence, etc.)
What kind of real evidence about the dangers of AI that are happening do you have that didn't come from an AI company, wasn't about some internal test that conveniently can't be replicated or independently verified, where nobody disabled guardrails that would be in place for typical users, or where they didn't instruct the AI to roleplay as someone doing something evil? What danger are we seeing right now in the real world that people aren't taking seriously?
The article is very dismissive of recent mathematical proofs. It mixes together ethical problems (plagiarism, "strip mining" proofs) with capability problems (proofs not being novel, overstating capabilities). Again, it's fair to be skeptical of what the companies themselves are saying (and why). But from what I've read, many mathematicians are taking AI quite seriously. I've recently read several thoughtful essays from credible folks who are calling for dramatic changes to the field in response. For example:
https://www.daniellitt.com/blog/2026/9/13/a-beginning-for-ma...
The article is also dismissive of the Hugging Face incident and other hacking incidents. This one is trickier, because we are more reliant on what the companies have released to piece together what happened. The METR report is probably the closest thing we have right now. Here again, I think the issue is that a lot of coverage blends the ethical issues (holding AI companies responsible for what their bots do) with capability issues (whether AI is or will be capable of developing novel and dangerous attacks).
That's not what happened, in any of the many incidents. But even if it were, if you can make AI do dangerous things by telling it to roleplay doing dangerous things, then that... Makes the AI dangerous.
> That's not what happened, in any of the many incidents.
That's exactly what happened when Google's AI "autonomously" hacked into companies coincidentally in the middle of tests of its cybersecurity abilities in a simulated highly controlled scenario on a totally isolated system that was just accidentally given unrestricted access to the open internet.
That's exactly what happened when AI was "threatening to blackmail" people. Internal tests, in roleplaying scenarios designed to push the model to do exactly wthat it did.
It's exactly what happened in the the Hugging Face incident. An internal test was conducted, AI was told to do tasks specifically to test AI's ability to take known vulnerabilities and turn them into working exploits, mostly using an internal research model which was not intended for production, but also another production model whose guardrails had been intentionally removed.
So yes, a company with billions and billions of dollars and years of research and investment into AI models likely will be able to use their AI in cyber attacks. I think pretty much everybody believed that was a risk.
What does that tell us about our risks in the real world right now? Well, it's extremely hard to say because we're still left trusting the AI company who is constantly lying about what their AI is to tell us what it was doing and how, and we're still unable to independently verify or replicate what happened, and there are no known examples of anyone who isn't an AI company doing anything like it. Certainty you or I or the vast majority of the population can't just browse to chatgpt.com and hack the planet.
Did AI do something that humans couldn't already do? No. Did they spontaneously "go rogue", or demonstrate "superintelligence" or "create AI civilizations"? No. Can we be sure they didn't just orchestrate the whole incident to push their "AI scary!" narrative? Nope. So we're pretty much left with what we already knew. All the largest risks are from AI companies themselves and governments building AI models. There isn't much any of us can do about that.
So, your claim is that OpenAI intentionally might have had its agents hack multiple companies, as well as multiple government agencies? And they're just pretending to not have known about it?
The vast majority of the AI-doomerism (if one wants to call it like that) actually doesn't come from AI companies, rather, from a few (high-profile) individuals.
(I intentionally ignore the tsunami of trash journalism and the likes)
"White-collar work, where you’re sitting down at a computer, either being a lawyer or an accountant or a project manager or a marketing person, most of those tasks will be fully automated by an AI within the next 12 to 18 months" https://www.youtube.com/watch?v=YTrBz6Z5c0E
AI companies aren't leaving a lot of AI-doomerism ground uncovered.
Best tip in here: when tweaking details like colors and fonts, just ask Claude to make you throwaway picker tools right in the site.
This is totally obvious in retrospect but I never thought of it (probably because of 25 years of thinking about tools in terms of reusable components).
There’s a nice Apple session from this year’s WWDC about this where they suggest using the new resizing feature of the simulator (born to support the foldable iPhone) to have a “drawer” for all the controls and tools to fine tune your UX. I’ll be stealing that for the web as well
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price.
Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!
But if they tried to set a fair price they would have to report how much money they are losing on each token sold. This might be bad for the real business of ai firms, hoovering up as much capital as they can
It’s not different from any other tool. If you use a dangerous tool recklessly, you should be liable for the damages. That means holding OpenAI liable for HuggingFace hack because they ran the tests, and the same goes if someone did something similar with GLM.
Of course in cases of negligence a tool maker could also be held partially liable. That’s a matter courts can decide. The main point is we shouldn’t jump to making special laws around the development of LLMs. The starting place should be enforcement of existing liability laws. New laws take time and will be heavily influenced by AI companies seeking a regulatory moat for their business. Moreover, it is a distraction from the illicit behavior that is already going unchecked.
My takeaway:
We should be not be concerned about AI bots' "goals", we should be concerned about the goals of the companies making them. Powerful but not sentient technology in the hands of reckless accelerationists is a plenty dangerous enough thing.
I mean, of course we should be concerned about the goals of the companies. But that's a second, separate concern and it's important not to muddy the two together into a single point. We shouldn't take any of these companies at their word and we should treat their stated intentions as suspect regardless.
The agents' "goals" in the specific instance being discussed is a benchmark. But it's also the case that at no point was the agent given the "goal" of hacking HF. The agent was tasked with solving a problem on a standardized test and it independently set a secondary sub-goal of cheating. And it nearly succeeded.
I don't think Cory argues against this. The idea that the agent nearly managed to succeed at cheating through a series of exploits remains true. But Cory does effectively call this unremarkable and uses some mental gymnastics to achieve that argument (somehow using the idea of an agent harness and how is written in "easy to master" Python as evidence), because the LLM is trained on hacker techniques.
I find this a dramatic oversimplification of absolutely everything going on here. Yes, it's bad that this happened. I think we all agree. Where I fundamentally think Cory is wrong is that this is a company doing bad things at the end of the day. And yes, I think OpenAI was irresponsible (to whatever degree). But ultimately that's missing the point: the incident points out that agents can do this without being explicitly told to, and more importantly, they can succeed at it. Slapping frontier labs on the wrist doesn't change the fact that this is possible with technology that exists today. It doesn't change the fact that other countries and companies are doing lord knows what. Or that bad actors are going to be bad actors regardless of what regulations you put in place. Making crime illegal is the wrong lesson to learn here, the right lesson is that we now live in a world where this happens and it'll continue happening, and we need to put on our thinking caps about how to keep our systems safe.
Ok, but do you really think this would have happened if OpenAI had a goal of never allowing an agent to do something like this? If a company making any extremely volatile explosives failed to take precautions and blew up a factory, shouldn’t we focus on their negligence?
Perhaps we might ALSO decide that this new explosive is so dangerous it needs specialized laws to protect the public, but that would seem premature if all available evidence indicated that the explosion would not have occurred if the company had taken the most basic precautions.
> Ok, but do you really think this would have happened if OpenAI had a goal of never allowing an agent to do something like this?
There's nuance to this problem. Is you read an indepth analysis, you'll find that agents acted in a very surprising way - it wasn't just a matter of loose guardrails.
More in general though, what you're describing is in essence the reason behind the call to pacing that some are pushing now. Problem is that we're in a race between countries that nobody want to lose - and indeed, Trump denied that there are such problems and refused to enact any measure.
Details the design and architecture of Apple Watch's new "Audio Intelligence" features. In particular, regarding Siri Recap which has been discussed elsewhere:
*How Siri Recap respects those around you*
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
Want to be clear that this is not me endorsing the feature. It's clear Apple did think about the privacy/social implications of the design, but it's also clear to me that there are real and perceived problems with having automatic conversation notes. Real differences from written notes:
- It was recorded and transcribed by a machine, not a human
- There are time stamps and locations
- You can't tell someone is using this feature (it is obvious when someone is writing notes during a conversation)
As far as perceptions goes, it feels very tone deaf of Apple to launch this feature in this way at this time. Read the fucking room guys.
Maybe this will be true legally (we wont really know until it is tested in court), but I think there's a huge difference in practice.
Siri Recap is an electronic record of a conversation that includes timestamps. Imagine for example that during a scheduled meeting with my boss, he threatens to fire me for not doing something illegal. I now have strong evidence that this was said (even if it cant be officially attributed to him) during the time I was in a meeting with him. Not saying this will hold up in court, but it certainly is more damning that having written notes (not to mention the fact that my boss probably wouldn't say this when I am taking written notes).
Making that feature illegal may backfire spectacularly across the whole IT sector. What Apple does is it inputs voice data into an AI program and that outputs some transformed output. If a court would rule that output in any way matches the input (to be classified as a "record") then the whole ethics principle of stealing other people's data and funneling it through AI to make it company's own, would be at risk.
I imagine no single spineless impotent modern court would risk a wrath of our new benevolent AI overlords calling them all criminals they really are.
(And to be clear: I’m not talking about being skeptical of fantastic claims about things that have not happened- ie an AI takeover, runaway super-intelligence, etc.)
reply