So it's OK to have more deaths so long as we can punish someone with no expectation that doing so will fix the problem? I suspect you are right, but I don't think it reflects well on us.
I think a way to think about it is that we (at a societal level) wouldn't accept a desk calculator to be sold as a consumer product if it did correct arithmetic almost all of the time but then would spit out a close but wrong answer 1 time out of 1000, even if that would be significantly better than the average person doing arithmetic. Our expectation would be that it could (and should) perform flawlessly. If our technological progress was such that the only calculator we could ever hope to produce would still have that kind of error 1 time in 1000, would it be unethical to prevent that from being sold? That's hard to say!
One of the heuristics built into us (because we're mortal beings living beings in a competitive, historically resource-poor environment) is that we trust the devil we know more than the devil we don't, and so unless there's a strongly compelling reason to trust autonomous driving devices a lot more than humans, there will be some inertia against using them, even if the calculations are that it will save X number of lives. I mean, inherent in that calculation is a level of uncertainty and people don't necessarily trust that number, because they don't have a reason to trust it, because they haven't really seen enough to trust it. Why take a company's word for it that it's safer when they have a financial incentive to do some creative stuff to get their marketing pitch? I would say that if you feel it doesn't reflect well on us, it's because it hasn't been thought about enough.
I think intuitively the desk calculator analogy makes sense, but from a technology perspective it does not. A calculator has a limited number of operations and operating modes, and you can feasibly test all of them, quickly, easily, and in an automated fashion.
An autonomous vehicle has an uncountable number of operating modes, and it is not feasible (perhaps not even possible) to test it in all possible conditions and states. Even if you could, doing so for (say) every single software change would take years each time.
Maybe that does mean that this is a fool's errand, and we just shouldn't be building autonomous cars, at least not until we have AGI that can think and act faster and with better judgment than a human.
I personally do think that "better record than a human driver" should be sufficient (perhaps with some significant, TBD margin; 0.1% better is probably not enough), but I accept and agree with your toplevel comment that sort of thing won't fly in the real world. The bar is really more like the self-driving car has to avoid making the specific kinds of mistakes and illegal/unsafe acts that a human driver would do (all while not creating new classes of mistakes that a human driver would not make), and, on top of that, be better than a human driver in situations where crash would not be deemed that human driver's fault.
I don't disagree that a calculator and a car are different things, but the important thing for the analogy is that the wider society expects the same thing of them. When John Q Public thinks about self-driving cars, he thinks to himself "How hard can it be? Drive the speed limit, stop at stop signs and traffic signals, go when lights turn green, stop at obstacles and stay between the lane dividing markers. If these vehicles can't do that then what good are they? Why risk letting those things on the road if I can't even trust them to do that?" Obviously you and I both know that the base things like "recognizing a stop sign" required decades of research to become reliable, but like, back in the 1960s before the problem had work started on it, even PhD holding computer scientists were thinking that object recognition was basically a done deal.
What I want to keep reminding the software developers with consequentialist ethics is that the entire rest of the world operates under a totally different mental framework than the techno-utopian one. In that mental model, a self driving car is an appliance and it will be judged by the standard of all appliances, which is that any catastrophic failures that happen during their normal operation are evidence of a defective, untrustworthy product. The problem they're trying to solve and its complexity has little bearing in that judgment. What incentive does anyone have to be generous and forgiving about potentially fatal errors when you're not taking the consequentialist viewpoint for granted? That's the thrust of my observation.
I was thinking about this and one way of thinking is what u said. We already allow sale of spaceships which have non-zero failure rate. So we don't necessarily need zero failure rate self driving.
But a problem with cars is, usually ur malfunctioning calculator just harms u, but a malfunctioning car will affect people who don't agree with ur choice of driving non-zero failure rate self driving car.
The rest of gen AI tools might prove that wrong- Need help with your homework? We have software that will only sometimes give you facts that are incorrect. The res of the time it does great!
