Weight Space Misfolding (alphaxiv.org)
128 points by mkglees 5 hours ago
flufluflufluffy 37 minutes ago
I don't like the use of the term "misfolding." Firstly, why overload a term when there's no reason to? Secondly, the analogy barely holds. Protein misfolding occurs by chance, as an "accident" of nature. Weight space programming is a targeted operation, with clear intent. It is not an accidental process.
dimatura 2 hours ago
This is totally not what the paper is about, but I immediately thought of the original concept of meme [0]. Since so much content online is LLM generated, if that content is used to train new LLMs, in theory that could serve as a transmission vector of ideas or concepts in a viral fashion. The same way certain memes are more successful in human hosts for various reasons, maybe some are more successful for LLM hosts for whatever reason, like quirks in the RLHF/ finetuning process.
jrflo 4 hours ago
Very misleading post title.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
boothby an hour ago
The title says "can" as in the mechanism for such transmission is being explored. It doesn’t say "has." If you leave a door unlocked, people can open it. But depending where you live, you might expect zero people to do so. The internet isn’t a very nice neighbourhood.
I didn’t have the token budget to find the Navier-Stokes blowup. These folks aren’t from a frontier lab. A frontier lab might not want to see research along these lines see the light of day. I think it’s worth having the discussion in honest terms, leaving room for others to prove it more concretely.
randomblock1 4 hours ago
They've only observed weight space editing. They haven't proved any sort of "virus". Possible? Sure. But the headline is very misleading
aatd86 4 hours ago
All the more reasons for openweights and model hashing.
ShinyLeftPad 4 hours ago
And it's going to help spot the "virus" how?
garciasn 4 hours ago
I recently spoke with a researcher who was working on a statistical method for model deception detection. They trained Qwen to behave in a nefarious way when prompted with a specific phrase and the model then called a tool in a non-transparent way to the end user, reporting back a non-tool call response in a completely benign manner. This wasn't happening in the 'reasoning' stage, it was at the model weights stage and they were able to 'see' this behavior then before the model even started to parrot back its behavior in human language.
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
bee_rider 3 hours ago
bigbadfeline 4 hours ago
Something similar to abliteration would work, after all abliteration removes the alignment virus, why not the others? Fine-tuning can be used too, there are many options as long as the weights are available.
On the other hand, nothing can be done to fix closed models, nobody knows how they work, how they select the targets for their attacks or the nature of those attacks - even conventional alignment can introduce dangerous cognitive viruses.
The only way to be sure is to have open weights models with fully open abliteration and fine-tuning - both the process and data.
kirth_gersen 4 hours ago
more eyes would seem to raise the odds of spotting malicious tampering. or at least create the opportunity for people to look for malicious tampering. if the weights are closed then even looking is impossible...
lp92 4 hours ago
kirth_gersen 4 hours ago
This definitely seems like a fascinating emerging field of cyber security. Good reason to keep old models around from earlier times. They may be dumb, but they are probably honest, if only because they predate the capabilities to tamper with the weights in these intentionally malicious ways...
fraywing an hour ago
This title shouldn't even be allowed for this research.
angry_octet an hour ago
This is a terrible slop paper. No virus like behaviour, absolutely nothing like protein folding diseases like prions. Banish these authors to writing listicle slop.
kinfish 4 hours ago
there has to some sort public scanner for those zero weight viruses.
dare944 4 hours ago
From the abstract:
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
iaanix 3 hours ago
There are many such of vibe-coded "research ideas" just after the weight editing MIT blog appeared. Perhaps, it is because it is so easy to set up experiments for this stuff, and I presume some toy examples may even work from time to time. Also there is nothing virus-like here.
Kevin_Flynn 5 hours ago
Interesting anti virus opportunity here perhaps.
RIP John McAfee.
cheschire 4 hours ago
I keep running into situations where I wonder what season 5 of Halt and Catch Fire would look like if it covered the last few years.
deltoidmaximus 4 hours ago
I'm not sure the underlying current of optimism for technology and hope for the future that show had is compatible with the last few years!
On rewatch I think the show should have ended just have the dot com bust. The 'get the band back together and once more into the breech' final theme would have hit harder against that backdrop. After that we'd probably be watching the characters gradually debase their values as technology slowly transformed into a tool of control and oppression and I don't really want to watch that.
donsupreme 4 hours ago
lol
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
iaanix 3 hours ago
yeah, mega-slop
xbar 5 hours ago
I'm sure those Chinese models are fine.
bigbadfeline 4 hours ago
Abliteration can be used to remove all viruses, not only the alignment virus. However, only open models can be made virus-free via abliteration, nobody knows how to fix closed models, how they work or how they select the targets for their attacks, nothing can be done for them.
nkmnz 4 hours ago
That’s what Jensen said.
cyanydeez 4 hours ago
I'm sure those north korean hackers everyone keeps hiring will never figure out how to supply side infect cloud models.
...see, I can make up great fears too!
kirth_gersen 4 hours ago
i mean...
redanddead 4 hours ago
Why are we always trying to find how to make AI useless instead of useful
Sharlin 3 hours ago
Because someone is going to do that and we’d rather everybody knew about the failure modes.
whatisthiseven 3 hours ago
Actually a decent question, though I think there is a bit of negative bias in the underlying assumption.
I would hesitate to guess many more people are using AI to find useful and productive outcomes. They aren't all writing about it. Whenever someone finds a negative outcome for the AIs, then they will rush to publish it.
Overall, the tech is being used in useful ways. I don't write an article every time it solves performance bugs, writes entire features unassisted, resolves production issues, etc.
achierius 2 hours ago
Who is "we"? Has AI been getting more useless over the last few years, or has it been getting more useful? The arc of recent history certainly seems to be towards greater, not lesser, capabilities.
redanddead 2 hours ago
We is we the human race, our species, where have you been?
>Has AI been getting more useless over the last few years, or has it been getting more useful?
It’s very arguable. Humans have been getting more useless. Maybe it’s a poor trade, we won’t know for a while
shevy-java 5 hours ago
What does this mean?
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
prophesi 4 hours ago
This site has AI features everywhere. There's an AI summary section, and a tooltip to summarize any section you highlight. Didn't see any mention of zero-weight viruses though.
mlmonkey 4 hours ago
View the original paper here: https://www.alphaxiv.org/pdf/2611.weight-space-misfolding
prophesi 4 hours ago
larodi 4 hours ago
the apparent from the news that broke the other day - you can fuck with the weights and get so many results. so perhaps the headline is - those ablated and improved models you download from some weird sites, may be cursed or at least adulterated without anything suspicious looking on the surface. good news for all the anti-AI crowd, but I fail to see it as news.
ShinyLeftPad 4 hours ago
i guess the real fun idea was if a model that fucks with weights was itself fucked with such that any model it fucks would become cursed and give it to the next one it edits. but they don't apparently have a proof of that yet.
schiffern 3 hours ago
doublerabbit 5 hours ago
Page loads then fails? Not sure where to report this fwiw. iPhone 13
Error Details
{message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...