Maybe? But I don't see why _necessarily_; your efforts can compound but each attacker will mostly be repeating work that the others already did. Defenders also operate white-box with instrumentation and relatively speaking have a lot more to spend. Consider how much attackers can extract from defenders (less than the defender's loss) and what this implies about rational spend.
I think there are a lot of unanswered questions like "is software security possible", in the sense of, will there be a long tail of bugs left for second tier models after frontier models have picked over everything?
Most predictions also sort of assume that the IT landscape looks similar to the way it does today. But there are a lot of possible futures, e.g. a more feudal model where smaller orgs without strong security capability shelter in the big clouds, "protected" by amazon, google, microsoft level resources etc.
The solution to this will be government and big corporate legal superiority. The widespread availability and ownership of guns in the US is a source of existential risk to the political establishment, but the normalization of said establishment retaliating with swift and overwhelming kinetic or legal force against any perceptible violent threat creates a powerful disincentive to would-be revolutionaries.
Correspondingly, the proliferation of high-speed digital offensive capability will probably be met with counter-hacking designed to identify and retaliate against such intrusions even faster or wider-spectrum attacks - a Gibsonesque cision of asymmetric cyber conflict. If 'black ice' doesn't take down an destroy intruders on its own, drone munitions probably will.
> Building yourself a house != becoming a landlord.
That's the point.
I have a landlord to handle things that break or need work. Yes, I'm paying to not worry about it but that price premium can be worth it if you value the flexibility.
How can you own your house and not be your own landlord? That's the entire point of the purchase. You take on all those responsibilities and all that work instead of outsourcing it.
i guess the real fun idea was if a model that fucks with weights was itself fucked with such that any model it fucks would become cursed and give it to the next one it edits. but they don't apparently have a proof of that yet.
I recently spoke with a researcher who was working on a statistical method for model deception detection. They trained Qwen to behave in a nefarious way when prompted with a specific phrase and the model then called a tool in a non-transparent way to the end user, reporting back a non-tool call response in a completely benign manner. This wasn't happening in the 'reasoning' stage, it was at the model weights stage and they were able to 'see' this behavior then before the model even started to parrot back its behavior in human language.
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
Did they (somehow, this seems extremely difficult) notice a pattern in the weights that indicated tampering, or just scan for the known manipulated weights? The former seems really useful and very difficult, while the latter seems less of both.
Something similar to abliteration would work, after all abliteration removes the alignment virus, why not the others? Fine-tuning can be used too, there are many options as long as the weights are available.
On the other hand, nothing can be done to fix closed models, nobody knows how they work, how they select the targets for their attacks or the nature of those attacks - even conventional alignment can introduce dangerous cognitive viruses.
The only way to be sure is to have open weights models with fully open abliteration and fine-tuning - both the process and data.
more eyes would seem to raise the odds of spotting malicious tampering. or at least create the opportunity for people to look for malicious tampering. if the weights are closed then even looking is impossible...
Your logic is exactly what the Linux people were saying. But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
> Your logic is exactly what the Linux people were saying.
There's nothing wrong with that logic.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Well, AI helps fixing bugs too and it can also write bug free code with minimal additional effort, so open source not only retains its advantages in the presence of AI, it multiplies them. After the initial (and artificial) hype and bug-fixing efforts things are going to settle down quite nicely.
Closed source, on the other hand, doesn't have the advantage of multiple eyes and minds finding and fixing bugs and the unavailability of the source isn't an advantage either - AI can decompile and understand binary too, so it can find bugs without access to the sources as well.
> Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
You're using one-sided logic again. If "no human understands how the weights actually impact the model" then viruses would be impossible. Apparently humans understand enough to introduce viruses and that's enough to detect and remove them. It's similar to alignment and abliteration, as I explained above.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
The source code being available doesn't mean there wont be any bugs, it only means that it'll be possible to find and fix the bugs without waiting whoever gatekeeps access to that code to do it for you.
Similarly, the weights being open, means that 3rd parties can potentially locate and fix issues with them without waiting for Qwen, ZhipuAI, Mistral, or whoever else to do it (who at that point might as well have moved on).
And yes, open weights wont prevent a model from being poisoned, just like open source doesn't prevent software from becoming buggy or exploitable. Where it helps is for 3rd parties being able to find, fix and release the fixes without depending on the original creators.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
That's by design though. It's just that there's now an exponential increase in the amount of "eyes", in the form of AI agents, looking at the FOSS code to find and fix the bugs.
After this much investment into the "AI race" the whole economy of US, pensions and everything, is indirectly tied to one company. What do you expect, the government will do something that makes the numbers go down?
She just happened to be in charge of a company that was buying from his and totally coincidentally had Alibaba as the largest customer and main financial backer and lender.
It's clear though. Even if an AI had connected all the dots on the obsolete shorthand, it would never have recognized the human value. I doubt that any AI process would choose to spend the energy on interpreting this particular obsolete handwriting and then understand that it was from the hand of a genius.
Using AI to do your job does not mean that AI suddenly walks in the door and escorts you out and starts collecting your paycheck.
Someone is still driving these studies, and probably will be for a while. Or maybe not. But still we can reward this guy for opening up this can of theorems that had before.
Silliest case:
Imagine a dusty basement terminal screen open to claude code with "There we go! Cancer is cured as you requested, let me know and I'll send off the results" and nobody is around to approve it.
At least that's how traditional Western universities used to work. The profs sometimes published something, and they taught students. Looking at science back then, it seemed to work out rather well.
We pretty much never only judged the output. An oil painting is valued because it's an oil painting among other things. If people know you have worked on something for a long time, it's valued more than something you just came up with this afternoon. A chair crafted by a woodworker, even if it's worse in some ways, is valued over an IKEA chair produced in a fraction of the time.
Generative models are like outsourcing to cheap mass production factory, not an advanced woodworking tool in your basement.
Both of those things are visible in the output. An Ikea chair isn't denigrated because we're such snobs, but because it's made of inferior materials. If Ikea started selling solid hardwood bookshelves tomorrow at the same price as their other stuff, I'd be driving right over there!
No, because you can get indistinguishable hand painted oil clones of famous paintings from Vietnamese rep shops and they obviously sell for far less than the originals.
Barring AI generated art and prose assets is valid, if for example you don't want to have the AI generation marker on steam, or you object to the use of AI for various ethical reasons. Also there's apparently quite the market of motivated purchasers that want to signal their anti AI values by buying from projects that don't use AI.
The IKEA chair costs less, but to say it’s valued less is too simple. Valued by whom? Only a small minority will ever buy a chair that was handmade. Presumably they prefer it, but they’re rare.
IKEA’s many customers deliberately chose the IKEA chair. We don’t know what other chairs they might have preferred if they were the same price, because they’re not.
Well that's the point a custom chair is valued more
I'd say most IKEA customers choose IKEA because it's cheap and they don't really consider or maybe even have an option to get a custom chair made, they don't know someone who could make one or they don't want to pay for it.
Hmm, I don't know that time invested inherently equates to quality. I would place FAR more value on a comfortable functional mass-assembled chair than some wobbly piece of crap some guy spent two weeks cobbling together in his garage.
It's not just time, it's also effort and other non obvious things like supporting a real human being
> I would place FAR more value on a comfortable functional mass-assembled chair than some wobbly piece of crap some guy spent two weeks cobbling together in his garage
If it's wobbly then sure, but what if it was made by someone with a modicum of expertise to make it not wobbly.
then again, i don't mind a good wobble, it's a fidget opportunity and i like to fidget
I suppose we're talking about "craft" vs "art." Making chairs is a craft. There are objectively "right" ways to do it. You can still be artistic about it, so long as the craftsmanship is sound. That being said, I will take a well-crafted chair over one that is not every time over the "artistic" one.
With regards to writing, I can't just throw word salad together and call it a day. But my writing style is my own voice - I pretty much write the same way I speak. With AI... well, after you've read enough of it, you notice that it, too, has its own voice. Like, I can distinguish the writing style of Gemma4 real quick. It smells like ozone.
I don't necessarily mind, so long as its telling me something coherent and accurate. You could argue it lacks originality, and it certainly lacks agency: But it IS usually well-crafted. I'll take it. Won't stop me from reading, say, PKD novels. But miss me with the artistry of, say, Finnegan's Wake, or even Ulysses, I'm apparently way too dumb for those (yes, I've tried). PKD would crank out multiple novels every year, Joyce took like decade to write incomprehensible nonsense.
it's not gonna save you here because it's very asymmetric and not in your favor, if you're a defender.
reply