I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.
It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.
I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?
I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.
This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.
Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.
So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.
Mine's never said such a thing, that's a good idea though.
I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.
I think the project difficulty grows at least 10x with all the dust, insects wind, pollen in the air and whatnot even assuming you hide it for bad weather and you have secured very stable mount points. It sure would be cool though.
The more I think about, I wonder if a small tethered ground robot paired with an overwatch camera would be better. Cable management could still be in the air but since it's just power delivery, it doesn't need the tension.
It would be interesting if you could just have all the motors hidden on the sides and have even the "cut" functionality driven by a cable getting pulled.
Does anyone know about how you could go about training this type of thing into the model? Surely it is quite tricky to get it to do such things without it doing them more than this one incident? It should have happened lots if it really was trying to obtain secret info and build backdoors?
reply