It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
> Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together?
Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.
This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.
I don't suspect there is a uniform spectrum between those two. I think this is something that's going to be clinal, which we see in a lot of other comparable social contexts. The number of people actually willing to cross a moral threshold into outright crime is relatively small, but those are precisely the people who cause the most damage when they get away with their behavior.
Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.
In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.
Your idea of a criminal seems to be hypercompetent and think of everything. These do exist, but most criminals are not very smart. Smart, dedicated, technical people can typically make more money legally.
Your argument applies to any imperfect security technology -- aka practically all of them.
> Your idea of a criminal seems to be hypercompetent and think of everything.
No, my idea of a criminal is someone who is motivated to commit crime, and I feel that we've already established in this thread that the approaches we're discussing are motivation gates far more than competence gates.
> Smart, dedicated, technical people can typically make more money legally.
Then who's been running all the botnets, writing cryptolocker malware, and running phishing scams for the past couple of decades?
We've always had script kiddies, and now we have people using AI itself to do malicious things. Technical skill has never been an obstacle for sufficiently motivated scammers.
> Your argument applies to any imperfect security technology -- aka practically all of them.
Ultimately, everything has weaknesses, and with enough effort, most measures can be circumvented. But how much effort is enough varies wildly between solutions.
There's a huge gulf between a "no trespassing" sign, on the one hand, and a concrete wall topped with barbed wire, on the other. The "no trespassing" sign only keeps out people willing to obey it; the concrete wall keeps out anyone who isn't willing and able to accept the time, effort, and risk necessary to climb over it or knock it down.
And the point is that using digital signatures to distinguish AI-generated media from hand-made media is much closer to the "no trespassing" side of things than it is to the wall. Maybe it's analogous to a gate with a latch you can open from the other side if you reach over in just the right spot.
I'm afraid it doesn't. To the contrary, treating effective security in terms of how it influences the attacker's cost-benefit tradeoffs, and evaluating proposed measures on whether the effort threshold they create is enough, is quite literally the opposite of black-and-white thinking.
And in this case we can clearly see that a solution that (a) does not substantially increase their costs -- and in fact, as I've pointed out above, only really filters by motivation, not by time, money or effort, and (b) doesn't target their potential benefits at all, is one that isn't likely to be effective.
You're conflating the effectiveness of the mechanism with its systemic impact.
* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.
* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).
* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.
* ssh: This is just an inappropriate comparison.
The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.
Pangram: I'm not arguing against encouraging skepticism; I'm arguing that the technology is not useless. It's good for people to know the limitations, but it's still evidence.
Antivirus: "Actually, we should build this hypothetical better thing" is a cop-out.
HTTPS: C2PA is a strict upgrade from unsigned photographs, so it should be used wherever possible.
SSH: Totally appropriate, the entire point of the discussion is whether it's permissible to the user that they might be more secure.
You're missing all of the points that there could be by focussing on random people.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
I am repeating myself, but this is about systems, and not about people.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors".
Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things.
Argh and I ran with your term aah
This is pascal's mugging. Democracy itself might fail! You're just inflating the stakes of your hypothetical bad outcome until it can overwhelm any positive upside.
Not to mention, democracy is under just as much or more threat from "banal" fake news created by citizens. People gonna people. They don't need the DPRK lying to them to fool themselves.
Yeah I think any additional security is good. At worst this could help in a lot of court cases. Someone presents photo evidence - it could be manipulated - it could be not. This happens already. Then someone produces an original higher quality version (like a raw photo - which I take even on my phone at all times now) and experts can verify that as the original.
And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.