I don't disagree that a calculator and a car are different things, but the important thing for the analogy is that the wider society expects the same thing of them. When John Q Public thinks about self-driving cars, he thinks to himself "How hard can it be? Drive the speed limit, stop at stop signs and traffic signals, go when lights turn green, stop at obstacles and stay between the lane dividing markers. If these vehicles can't do that then what good are they? Why risk letting those things on the road if I can't even trust them to do that?" Obviously you and I both know that the base things like "recognizing a stop sign" required decades of research to become reliable, but like, back in the 1960s before the problem had work started on it, even PhD holding computer scientists were thinking that object recognition was basically a done deal.
What I want to keep reminding the software developers with consequentialist ethics is that the entire rest of the world operates under a totally different mental framework than the techno-utopian one. In that mental model, a self driving car is an appliance and it will be judged by the standard of all appliances, which is that any catastrophic failures that happen during their normal operation are evidence of a defective, untrustworthy product. The problem they're trying to solve and its complexity has little bearing in that judgment. What incentive does anyone have to be generous and forgiving about potentially fatal errors when you're not taking the consequentialist viewpoint for granted? That's the thrust of my observation.
What I want to keep reminding the software developers with consequentialist ethics is that the entire rest of the world operates under a totally different mental framework than the techno-utopian one. In that mental model, a self driving car is an appliance and it will be judged by the standard of all appliances, which is that any catastrophic failures that happen during their normal operation are evidence of a defective, untrustworthy product. The problem they're trying to solve and its complexity has little bearing in that judgment. What incentive does anyone have to be generous and forgiving about potentially fatal errors when you're not taking the consequentialist viewpoint for granted? That's the thrust of my observation.