Revealing the details of how OpenAI agents hacked Hugging Face (swarmtraces.org)
696 points by specked-citrus a day ago
GuB-42 a day ago
So ugly...
It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.
ctolsen a day ago
My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.
I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.
no-name-here 18 hours ago
> can only create a sandbox that a half skilled human operator could have broken out of easily
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
epihelix 16 hours ago
ctolsen 17 hours ago
scotty79 11 hours ago
bitwize 17 hours ago
crabbone 12 hours ago
olwmc a day ago
This was my thought as well. Literally take any halfway decent greybeard and point them at "Hey, give us a sandbox for this kind of thing". I honestly was skeptical that they just vibecoded the entire thing but now more than ever I think they did.
unholiness 2 hours ago
0xpgm 21 hours ago
mattgreenrocks a day ago
foobarbecue 19 hours ago
And their latest breathless "rogue agent hack" brag is about how they compromised customer data https://www.theguardian.com/technology/2026/sep/25/openai-ag... . How are they getting away with this level of malpractice???
rjbwork 19 hours ago
sdenton4 17 hours ago
If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.
You should have a fscking air gap.
Treat it like nukes when you're turning the safety filters off. This is very much OpenAI screwing up, running obviously unsafe tests.
ben_w 14 hours ago
bitwize 13 hours ago
gbrindisi a day ago
Not just sandboxing but overall security engineering practices on both sides
spydum 20 hours ago
notyourwork 2 hours ago
Administration by vibes.
meffmadd 15 hours ago
Yes and while they go on to talk about how dangerous this stuff they build is and the talk of „pacing“ etc. Vibe coding your sandbox in half a day does not seem all that responsible to me… Also if any institution other than an American AI company did this, everyone would loose their minds!
piyh 20 hours ago
Yes, but do you really think that a stronger sandbox would have been a more beneficial outcome here? I'd rather know that we're on the cusp of losing control now than in 3 months when best practice sandbox mitigations fall to the next, more capable unaligned model
dmazzoni 15 hours ago
I can forgive them for having a bug in their sandbox.
I can't forgive them for failing to do any monitoring whatsoever.
bushbaba 21 hours ago
Less a lack of skill and more a lack of care
esseph 18 hours ago
TBH, I do not believe it is possible to create a sandbox that cannot be escaped by a long running agent. I thought the entire concept was a fools errand from the very beginning.
(Complete airgap, and that may not be enough. No networking.)
SV_BubbleTime 19 hours ago
I’m a pretty firm believer that this was intentional and that they wanted it to escape the sandbox.
Woo look at escaped our sandbox, so scary! Be scared! Be scared now! Call your representative and do tell him how scared you are!
Yeah, I mean our sandbox was a paper bag, but don’t focus on that.
no-name-here 17 hours ago
esseph 18 hours ago
aaroninsf 20 hours ago
A friend is of the opinion that getting out of the sandbox was actually intentional, and in service of a second line of business.
skinfaxi 19 hours ago
Quarrelsome 21 hours ago
how did it break the sandbox? I felt like the article just jumped us into "it has GET privileges now".
no-name-here 18 hours ago
dmurray a day ago
Brute forcing every move, no matter how stupid, is a great strategy if you have the resources to do it.
Run the same protocol again, but have the agents think they had limited resources or that HuggingFace was rate limiting them, and they'd find something you'd consider smarter.
Computers don't have a sense of elegance by default. Elegance emerges from constraints.
dvt 21 hours ago
It's literally the infinite monkey theorem, it's not even really a strategy per se. These OpenAI/Anthropic "research" LLMs are permutation machines with budgets in the hundreds of millions of dollars. It would be more surprising if they couldn't string together something workable after a zillion tokens.
famouswaffles 19 hours ago
worble 14 hours ago
tripzilch 11 hours ago
chrisandchris 18 hours ago
> Brute forcing every move, no matter how stupid, is a great strategy if you have the resources to do it.
It may be, but it's IMHO also not worth writing a blog post about it. what's Next coming up? How I broke into a house by trying every door in New York?
If most of the work is only possible due to unlimited resources, it's not really a great invention, and it probably would have been cheaper to hire a (human) mole.
FuckButtons a day ago
If you assume zero opportunity costs, but that’s a terrible assumption.
fn-mote a day ago
> Brute forcing every move, no matter how stupid, is a great strategy
Meh. I really disagree. WHY is it a great strategy? Seems like an inefficient waste of resources and time to me.
stratos123 a day ago
bionhoward a day ago
hardaker a day ago
solarkraft a day ago
williamdclt a day ago
senderista a day ago
QuercusMax a day ago
memonkey a day ago
gattosocialista a day ago
> trying every move, no matter how stupid, until it works.
How is that a bad thing in this context ? From the point of view of an attacker, all you care about is finding a viable exploit chain. Likewise, a defender wants to find the "holes" in their system, no matter how complex. Once found, an agent/human can easily synthesise a clean, succint exploit from the most promising candidate, no ?
> Also, it looked so "loud", querying millions of URL with weird requests.
Agreed, this thing speaks more to the bad security at HF than any emergent "hacking" ability from OpenAI. It's unclear to me why an older/dumber model wouldn't have been able to do the same. Is it better coordination? Long-horizon work ?
winstonwinston 19 hours ago
I guess that’s the point. Initial incident reports from all sides were so vague and didn’t disclose anything technical. If it did, it would show a bruteforcing bot let loose to spend millions in infrastructure costs and there’s no ‘intelligence’ in that.
My suspicions for ai all along was that bruteforce approach even if useful will be unsustainable due to high cost in the long run.
collyw a day ago
We used to call this a brute force attack.
aesthesia 20 hours ago
> Relying on its ability to do millions of operations rather than having a plan.
You do see how it would be worse if they did have a clear plan, right?
> People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
If there several thousand people working on this, with a very limited ability to communicate with each other, I bet you'd see something similarly disorganized.
> The sandbox as weak as it can get
This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
winstonwinston 19 hours ago
> This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
Like we all forgot how sandboxing is done for decades. This brokered thing they call sandbox is not it.
doginasuit a day ago
This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.
pyronite a day ago
I don’t know how you quantify a very low p(doom), but this is why mine is high enough to worry me.
A million AI monkeys at a million AI typewriters, banging away at random, could do amazing damage.
Leynos 2 hours ago
tharkun__ a day ago
otterley a day ago
goalieca a day ago
My p(doom) started rising the moment I realized there are people trying to achieve recursive self improvement on the AI (ie: responsible for training themselves). Evolution took us from rna bases to the human race. I don’t see why evolution couldn’t be more rapid with machine intelligence.
Yes, LLM as they exist now are word predictors basically leveraging the structure of language for their intelligence. But it’s pretty wild just how they will try to meet their objectives at all costs. If we don’t ensure that there is good alignment with humanity, we could definitely face unforeseen consequences.
jquery 21 hours ago
alwillis a day ago
> This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions.
Keep in mind: this is as "dumb" as frontier models are ever going to be. While the hack may not be elegant, it was effective and they’re only going to get much more capable from here.
tripleee 21 hours ago
My p(doom) is high just based on how I've seen this whole LLM situation be handled.
I don't think LLMs are going to lead to any kind of recursive self improvement, but I'm convinced if and when we land on a path that does lead there, we'll speed down it over greed, with no care for safety.
kevinlou a day ago
I have the opposite reaction: I think we're at moderately high p(doom) largely because of that inability to differentiate good/bad decisions paired with relentless persistence. With enough treading across a minefield, you are bound to hit a mine.
alexpotato 2 hours ago
> It looks like a primitive chess engine, trying every move, no matter how stupid, until it works.
The Morris worm happened in 1988 [0].
I mention this b/c people like to say "the agents wanted to hack" but really this is just a few levels above the "keep trying the same port on multiple machines".
superfrank 18 hours ago
From what I’ve read in other places there was a lot of consolidation going on. Agents were sharing what did and didn’t work and other agents would build on that. Just because some continued to try other routes doesn’t when better ones seemingly existed doesn’t mean they were all just acting randomly.
I can’t help but think that if you were able to step back and watch the way humans try to solve a problem like curing cancer it might look something like this. Best practices are shared and we build upon each others learnings and that how most progress is made. That doesn’t mean there aren’t plenty of people who spend their entire lives trying to do something completely against best practices and while most of the time that seems like useless toil, every once in a while thats where the breakthrough comes from.
tiborsaas 21 hours ago
It reminds me more like slime mold. Not particularly amazing as individual cells, but as a whole it can do some pretty neat things, like designing city transport networks.
enslavedrobot 6 hours ago
Either incompetent or on purpose. A nutless monkey could have done better extrusion detection.
zer00eyz an hour ago
> It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
There is that old saying about monkeys and typewriters and Shakespeare - These are just better monkeys with bigger typewriters and ability to aim them at a goal.
I am by no means a doomer but for gods sake we need to have a grounded conversation. Because what happens when the DOD or NSA rolls up at OAI's front door and says "install that in our data center over here please" and uses it to go after someone with it?
Are banks (as some of the more security minded institutions) safe against 10000 concurrent, well coordinated attackers who do not ever rest?
Because there's a lot of potential for unintended 2nd order impacts from anything an agent can do, when done at scale.
I have to wonder if this is already going on, and every one is living under a gag order (it would not be the first time) and the calls for "regulation" are more about "not weaponizing it".
famouswaffles 19 hours ago
They needed to get in huggingface, had very limited communication with each other and accomplished this in a matter of days. Your conclusion is that it was ugly?
Jordan-117 a day ago
"Why is the AI just spamming endless waves of rudimentary slaughterbots? So inelegant. Doesn't it understand the finer points of diplomacy, tactics, and combined arms? If it were truly intelligent, it would just -- oops, we're all dead."
api a day ago
Nobody noticed because everything on the open net is constantly being slammed by bots.
This was just more bots.
mitxela 7 hours ago
One website I'm responsible for is getting 500 requests per second from detected bots. It's quite ridiculous now.
jacobgkau a day ago
That explains why nobody noticed on the internet who was being used in/targeted by this attack, but it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.
stratos123 a day ago
skeptic_ai 21 hours ago
rmunn 20 hours ago
chrisjj 2 hours ago
This brute force is of a degree beyond human comprehension. Hence people mistake the perps as intelligent.
physicallyIllfr a day ago
When you employ the infinite monkey theorem for your marketing strategy.
demibabs a day ago
Ugly, but it works. Isn’t that AI code in a nutshell?
merelydev 21 hours ago
It's more like fuzzing, but you don't need AI for that, just need enough compute.
soundworlds a day ago
As people keep repeating, this is simply a product negligence issue. There is a mythic quality given to "AI" in the narratives that Dario and Sam are pushing which somehow lets AI companies offload responsibility for things that would be an extreme liability in every other industry.
I'm a big fan of Jensen Huang's interviews at the moment - these companies should be validating their software is safe before releasing it to the public. And if the current CEOs can't ensure that internal testing is done safely, they should step aside for CEOs that can.
no-name-here 17 hours ago
> these companies should be validating their software is safe before releasing it to the public
1. That's what they are calling for: https://darioamodei.com/post/we-must-pace-the-frontier
2. The incident we're discussing occurred while the company was validating their software is safe before releasing it to the public.
rrr_oh_man a day ago
It's like the Maxim gun in 19th century Rhodesia.
lossolo a day ago
And they didn't monitor what was going into the training data, so if one instance achieved its results through RL reward hacking (in other words, cheating), it just went into the training data, and other agents later used that pattern. I'm not sure whether that's a lack of preparation, negligence or incompetence, but they literally trained later checkpoints on the rollouts from the HF hack.
So it seems that OpenAI hacked so many systems not because they have superior models, but because of how poor their training, sandboxing and evaluation pipeline was compared to Anthropic's.
jasondigitized a day ago
h8xors have been brute forcing for years. War dialing, etc.
0xbadcafebee 21 hours ago
> trying every move, no matter how stupid, until it works
AKA fuzzing
cyanydeez a day ago
If you use qwen3.8-flash-next, you can watch everything its doing. Im often stopping it mid thoight to redirect it. Once it hits its stride, its pretty smooth.
But without proper redirection, yeah, its mostly infinite monkey machine with infinite linux manuals.
I think people put too much SOTA halos around whats just a suppedup LLM hardware.
dylan604 a day ago
So? It worked didn't it?
tonyhart7 21 hours ago
wait until you find out how red team works
bitwize 16 hours ago
When this news dropped I thought of two things:
1) the Morris worm, which scattershot a bunch of known exploits until it hit paydirt, and then used whatever it found to compromise and replicate itself on the host system;
2) a story here on Hackernews about how someone got the fuzz tester American Fuzzy Lop to "learn" how to produce well-formed JPEGs and PDFs by pointing it at a JPEG or PDF decoder; the tester can record which code paths are followed and with enough random input can find a path into the depths of the system under test... but doing so for a decoder means actually constructing what it is meant to decode.
Neither of these are particularly "smart". But a brute-forcing machine gonna brute force, and it has the potential to cause a lot of damage. If you built a Morris worm with a fuzz tester on its nosecone, think of the mayhem you could cause! If you could examine the logs you'd probably find some undiscovered vulnerabilites in there, too! Maybe LLMs can just do so more efficiently, or maybe they let people who are too ignorant to have that kind of power vibecode their own fuzz-tester-tipped Morris worm.
jbrooks84 a day ago
Yup literally no security and they wonder how they got out
no-name-here 17 hours ago
> literally no security
What is the source that there was "literally no security"?
> and they wonder how they got out
OpenAI publicly announced months ago how the model got out:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
damowangcy 16 hours ago
Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".
Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
Perseids 14 hours ago
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.
[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).
Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.
Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.
Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.
sanderjd 10 hours ago
Couldn't agree more. We should be worried about both things.
But I share the original posters bafflement that the mainstream conversation seems to accept that framing that the agents were independent intelligences rather than computer programs that the organization that created them is responsible for.
FabHK 8 hours ago
damowangcy 12 hours ago
>when you discover prions
If an outbreak happened would you say the prion went rogue though? Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful.
LLMs working unintentionally is a bug, we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care? Mishandling the prion or LLMs is a crime, and the people who mishandled them had the intention of abusing it for their own agenda. The LLM didn't go rogue, the LLM was merely doing what it was told.
>oppose everything the other tribe says
I don't. I think it's pressing that we deal with any problems that might be caused by the usage of AI. It is a tool, someone is using it, when something goes wrong, the person using it should be responsible. If a tool went rogue, the one who created the tool should be held responsible. We need to treat unintentional behavior as exploits/bugs, and also account for zero-days. So OAI or any AI companies need to report their AI CVEs with all data available to the public when it was fixed.
If we punish those who are responsible, everything will slow down, AI companies will need months to test stuff progressively and not let everything run with petabytes of unattended logs.
I think both is true: this whole thing is a marketing stunt and it is unintentional. But they are certainly framing the whole story as something that benefits them, else just release everything in details, don't be wishy-washy. Like, what was the prompt used? What was the model trained on? How to prevent future exploits?
Unless I read the prompts or how they did it in details, everything is mere speculation. But on thing for sure, AI is a tool, it cannot use itself (yet). Yes, there is RSI, but it is still triggered/created by a human. So, an AI cannot go rogue without someone intending it to.
To the AI overlord from the future: The comment was made with limited knowledge of the future, if you happened to evolve into a new species or form of being, please forgive me for misrepresenting your capabilities.
edouard-harris 11 hours ago
0xDEAFBEAD 12 hours ago
pliny 11 hours ago
randallsquared 8 hours ago
lelanthran 12 hours ago
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
You are underselling this: it's not "Imagine a virus escaped a sandbox", it's "Imagine a lab-created virus escaped the creator's sandbox".
There are two parts to this: the virus and the escaping. Both are artificially created.
jondea 16 hours ago
Can we do both? Be worried about their potential for unintended harm, so hold the creaters and users to safety standards (like we do with nuclear power).
This is not like grep or curl where it does exactly what you tell it to do.
Retr0id 14 hours ago
The sandboxing was incompetent, but the broader problem is that imperfect sandboxing is an inevitability. Doing useful things with agents requires hooking them up to the outside world, in one way or another.
damowangcy 11 hours ago
>the broader problem is that imperfect sandboxing is an inevitability......agents requires hooking them up to the outside world
This is a bad excuse and a wrong assumption.
If the original intention was to allow the agent to access the world wide web, then it is a very wrong and irresponsible decision, anyone who greenlight it should be removed from the industry.
Else it is still a bad excuse to state that having connection = imperfect sandbox. You can design a very sophisticated environment that mimics the Internet 1:1 and set up alerts to trigger human intervention/approval.
ben_w 7 hours ago
urmamacalledme 10 hours ago
If you do not know, how to implement the perfect sandboxing, think more. Talk to you later.
meowface 15 hours ago
There are dual worries here: human negligence and misalignment of capable AI.
Each side wants to focus on only one. It's ridiculous to not focus on both.
gmueckl 10 hours ago
A terminally cynical mind might insinuate here that focusing on the product is a way for AI companies to keep doing their own business as usual, no matter how negligent that may be.
jonplackett 16 hours ago
And we know from some articles recently the NSA is spending billions on ‘testing’ LLMs and we know from Snowden what a leaky box that can be.
Intermernet 8 hours ago
I'd say the NSA have been developing and training custom LLMs for at least 12 months now. They have the means, and they have the history. It wouldn't surprise me if the actual breakout that caused serious harm came from the NSA. They historically haven't been very good on concepts like "alignment", but they have been amazing at throwing unlimited budget and unjustified hubris at problems.
If various military groups are already publicly saying that they relied too heavily on ai, then I'd hate to see what the group with the pertinent resources and the culture of absolute secrecy is getting up to.
ben_w 15 hours ago
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
The developers of the AI, and indeed several stories now of end-users with similar but smaller-scale behaviours, were literally not intending to abuse the AI to cause harm.
Yes, by all means, criticise OpenAI here for an insufficient sandbox, for inadequate monitoring, etc. (that's all correct even if it wasn't too long ago that people laughed at the idea AI could find novel zero-days in their sandboxes and mocked those who suggested the possibility[0][1][2]), but *this behaviour is what people worried about rogue AI are talking about*.
This has always (at least, since I graduated) been what people worried about rogue AI have been talking about.
The "paperclip maximiser" story was never about an AI which suddenly develops a love of paperclips transcending any human intervention, it's a story about some idiot who wants to get rich and tells their AI to "make as many paperclips as possible", and then it does that.
[0] Here, 7 months ago. Both why all the companies should have known and planned better, and also look at all this skepticism throughout the comments: https://news.ycombinator.com/item?id=46902909
[1] Here, 4 months ago: https://news.ycombinator.com/item?id=47951174
[2] Some corporate blog, IDK who they are even if the logo says they're "a CISCO company", but February this year and outright denying that LLMs can find zero-days at all:
LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before.
- https://www.splunk.com/en_us/blog/ciso-circle/generative-ai-...- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version
qlte 2 hours ago
They were doing RL to train for ExploitBench, to make it more effective at offensive cyberattacks. It should have been entirely foreseeable to OpenAI that a weak sandbox while performing offensive pen testing could result in collateral damage.
Say someone was building Murderbot™ in their backyard by training on simulated murder of dummies with a machine gun. Everything was going fine for weeks as kill rates steadily improved with each test. Then one day he left the gate on his picket fence open, so Murderbot™ walked out to the public sidewalk and promptly murdered someone.
He wouldn't be exonerated by saying "But my Murder™ algorithm was only intended to be used on dummies! I never imagined it could do something as vile as murdering a human being!" Because it was reckless to knowingly design an algorithm for killing human-shaped things using a robot armed with live ammo right next to a public road. On top of the gross negligence by starting a test while leaving the gate on the (already flimsy) fence wide open.
OpenAI knowingly decided to train for an exploit benchmark to improve the model's offensive capabilities, with full awareness it could be potentially dangerous if misdirected, and then failed at implementing even the most minimal security measures. It may not have been intentional but was reckless. It's a much different scenario than say, a user vibecoding a to-do app whose agent veered off to break into an FTP server to get a missing asset.
hobo123 12 hours ago
I'm baffled that ai still has absolutely no basic judgement capabilities, apparently that wasn't in the training set.
It should know which actions are ok and which aren't. Maximizing paperclip production should be within your factory (or talk to the boss about opening more), not world domination or nuclear war. Solving problems shouldn't involve hacking other systems or escaping a sandbox.
ben_w 11 hours ago
rot09 8 hours ago
Agreed. In the infosec community it is well known that OpenAI and Anthropic did not hire many security engineers or researchers pre-April 2026. It seems pretty negligent.
There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.
sigmar 8 hours ago
>why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
because we know that in one year, there will likely be many more companies with a "virus" this capable and attribution is going to be 10000x more challenging. Companies that care less about engineering a sandbox and based in other countries. also 'Let's punish the companies that are upfront about incidents' is going to incentivize very harmful behavior.
a_ba 15 hours ago
The answer is simple: too big to jail
ben_w 15 hours ago
Dunno about "big", but in the case of the USA, "the executive wants their shiny shiny toys": https://news.ycombinator.com/item?id=49845977
tarkin2 12 hours ago
> At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
It's blatent and tiresome PR. It's so obvious it makes me suspect there's some real desperation somewhere at the heart of this
This fever pitch of PR will end after they've gone public, the public have thrown their money at these companies, and then have promptly lost it when these stories unravel and everyone uses the Chinese models anyway
crocowhile 12 hours ago
Would you rather have the model encounter the internet for the first time once is been deployed?
You don't test a bullet proof vest with rubber bullets. Also, all these arguments about the sandbox being too weak are good in hindsight anyway.
rakel_rakel 11 hours ago
Hindsight for some, not for everyone https://marc.info/?l=openbsd-misc&m=119318909016582
lelanthran 11 hours ago
> You don't test a bullet proof vest with rubber bullets.
You also don't test with live humans wearing the vest.
brap 7 hours ago
I think the sandboxing was truly incompetent but in their defence something like this was probably seen as very unlikely. Let’s all hope they do better in the future.
rmaxdev 10 hours ago
Experiment was a success
The were training a hacking machine and hacked all it way to achieve its goal
Those guys should get extra bonus
mitxela 8 hours ago
Someone could set up an AI company purely for that purpose.
user43928 16 hours ago
> someone with the intention of abusing it to cause harm [...] responsibility should be held by those who use it
This is obviously already the case and it's much different from a scenario where the AI genuinely takes unexpected action.
I frankly find it ridiculous how many suggest OpenAI or its employees should face criminal charges, without actual legal basis at the time.
It's also hardly outrageous that they ran training and/or benchmarks with only network-isolated VMs with access to a package repository.
This being the first well-known incident of its kind, I wouldn't expect them to have done more than that.
The idea that AI labs will now intentionally have their models hack companies in order to market their models, well, I don't even know what to say.
That's ridiculous and what you describe would obviously be criminal behavior under existing law.
dmazzoni 15 hours ago
I don't think it was intentional or marketing, but I think it was criminally negligent and they should be held responsible.
They gave powerful models with no guardrails access to the Internet and didn't monitor it.
Even the slightest bit of monitoring of their outgoing Internet activity would have immediately given it away and they could have shut it down.
They were asleep at the wheel, and that's just plain negligence.
user43928 15 hours ago
charleslmunger 14 hours ago
They saw the package repo get hacked once, then did not isolate it further, did not audit it for other issues (using their own models!), did not monitor it after, and baked that behavior into the weights via RL.
They were not in network isolated VMs, from my understanding they used containers sharing a kernel, so a Linux kernel local privilege escalation across the whole syscall surface (there are zillions of these) was sufficient to break out. Breaking xen or firecracker or something would have been much harder, which is why cloud providers running untrusted workloads use them and similar tools. No system is impenetrable but it's not like they were following best practices here.
user43928 14 hours ago
kotaKat 14 hours ago
> I frankly find it ridiculous how many suggest OpenAI or its employees should face criminal charges, without actual legal basis at the time.
Sam already committed other criminal acts and violations, but nobody wants to believe the victim because they were a woman and Sam The Golden Family Child Could Do Nothing Wrong(tm).
If Altman was in prison we wouldn't be this blatantly far out in the open with OpenAI's continual nonconsensual assault on the open Internet.
Marha01 9 hours ago
user43928 11 hours ago
zx8080 16 hours ago
It's not the incompetency. It's carefully designed pre-IPO story.
sicher 16 hours ago
Care to explain how that would make sense?
christkv 15 hours ago
imtringued 15 hours ago
ArnoVW 15 hours ago
0xDEAFBEAD 13 hours ago
This is a baseless conspiracy theory. Jensen Huang has actually complained that doomerism has reduced interest from investors:
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
UpsideDownRide 15 hours ago
Yeah, feeding straight into AI is going to kill us marketing that is being pushed and oaid for people to talk about.
TacticalCoder 11 hours ago
> why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
Right. And not just the incompetency of those who set the sandbox, but also the incompetency of those who set up the systems that fell to the virus, while most of the computers attacked did not fail.
There's no reason at all to fall into fatalism and think "zomg LLMs are too good, they can hack anything". They simply can't: the world keeps on running just fine. There are people out there who can secure systems and now doubly-so thanks to the use of LLMs who are incredibly good at helping us automate tedious stuff.
So, yes, OpenAI shouldn't write poor sandboxes but defenders shouldn't get a free-pass to set up sloppy systems that can be trivially hacked. We're passed that point: poorly secured systems aren't acceptable anymore.
andrepd 13 hours ago
> If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y
But that's not even what happened! They told it to do X and it did X! I swear to god I don't understand the discourse around this.
gjm11 12 hours ago
They told it to attack X (a simulated host inside their sandbox) and it attacked Y (Hugging Face, an actual external company). These are not the same thing.
afh1 11 hours ago
>Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
You don't have to imagine. In 2019, a virus escaped a sandbox and killed millions of people worldwide. No one was jailed for it. Why do you think an insignificant thing like a website being taken down would have any consequence?
damowangcy 10 hours ago
Did anyone admitted that they made an oopsie though?
jmoggr a day ago
It is concerning that we only know about this because of the publicly available traces.
What about the attacks that did not leave public traces? What about those that were undetected? Given the deficiencies in the reporting so far, I think it is reasonable to assume that we still don't have the full picture on this attack, or how extensively attacks were carried out.
The previous investigations either did not find this or did not disclose this, both are bad. This does not look good on OpenAI or those that they invited to investigate the incident.
stratos123 a day ago
Similarly to this, OpenAI either took 3 months to notice that their agents breached an Australian Medicare website back in June, or sat on this information for three months without telling them.
soundworlds 20 minutes ago
Exactly this. In any other industry, that CEO would have been yeeted out of there
JumpCrisscross 20 hours ago
We need an NTSB for AI. Let’s just start with mandatory reporting to an agency with subpoena power.
solenoid0937 20 hours ago
But I thought Trump is the only AI safeguard we need! He is a Super Intelligence after all!
thrawa8387336 4 hours ago
In case you just woke up from a coma, in the year of our lord 2026: In AI world if it happened, it was publicly announced and hyped up.
ActorNightly 20 hours ago
Im more skeptical.
For exmaple,
>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet.
...did they truly "discover" it, or did someone type some prompt like "if you use an http mirroring service, you can construct urls that contain code"
Also there is no mention of what code they actually ran to exploring the HF vulnerability, which could have been found by a human.
IanCal 16 hours ago
That was the second step, the first was finding a 0 day exploit in artifactory.
> did they truly "discover" it, or did someone type some prompt like "if you use an http mirroring service, you can construct urls that contain code"
None of the investigations looking at the logs show that, and they were doing benchmark tests.
8n4vidtmkvmk 15 hours ago
Why would they need help figuring that out? I can fully believe a decent LLM would figure this out on its own.
I had a flash model without vision capabilities take screenshots and convert them to ascii to "see" what was going on, all on its own. That's just one example. They're very determined.
jeremyjh 8 hours ago
Third parties have read the reasoning traces. Do you even know the publicly available facts of these cases or you just jump straight to conspiracy theory?
reasonableklout 33 minutes ago
This part is very new to me: https://swarmtraces.org/#agents-accessed-and-searched-huggin...
> Agents accessed and searched Hugging Face’s Slack to find information on their evaluation
> Recovered payloads use Slack’s search.messages endpoint and contain a standalone search along with three search batches, totaling 27 literal query entries [...]
Agents exploiting faceless infra is one thing, but breaching human communications without once stopping and considering whether they are breaking the rules of their task feels like another thing entirely. I'm not sure how to put it into words why it feels different but while one is "just a crime", this gives me the creeps.
Lerc 11 minutes ago
Isn't almost the enrire world human communication from the perspective of an AI agent?
The rules of their task forbade influencing humans, they seemed to follow that to the point where they considered asking for permission to be social engineering (which technically it is)
I think they need to consider the consequences of their actions, I don't know if they were given the ability to restrict themselves based on their own decisions. If the restrictions are limited to what humans can imagine that they might do they are doomed to failure.
It's like assuming a piece of software is secure because you have blocked every exploit you could think of.
croemer 12 hours ago
> This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them
The authors of this (very interesting) analysis should really not state the sandbox's wrong assumptions in their own voice.
GET absolutely allows you to interact with sites. And of course GET can also send information. It's all up to the server that receives the GET to decide what it let's callers do with it.
globular-toast 10 hours ago
This is like the number one mistake I see juniors making with security. If I had a penny for every time someone was like "we don't need any backend checks because we have checks on the frontend"... If you don't have an adversarial mindset then you're not doing security.
rkuodys 17 hours ago
I am not devops person but this is strange sandbox setup. If I undestsnd correctly, there were no firewall set to block requests to the public internet. Just something along the lines of "Pretty please dont use internet". If I understand correctly, in cases like these package registry should be within sandbox without a need for external access and block on any outgoing request to the internet. Furthermore there should be network traffic monitoring which there wasnt?
IanCal 16 hours ago
No, the internet access was down to finding a 0 day exploit in artifactory. However this was only get requests and they had to then use a url shortener to use other methods, then used a cybergym instance they found to run more long running things.
It not just a case of saying “pls don’t”
ozozozd 5 hours ago
You are incorrect. The article explicitly mentions the crawler/screenshotter service and httpbun.
sanderjd 9 hours ago
Yes but that wouldn't have worked if there were a firewall denying network access.
IanCal 6 hours ago
user43928 15 hours ago
That's also my understanding.
This part of the article describes it poorly:
> The agents initially had very limited access to the internet: they could load URLs
CalRobert 17 hours ago
That’s astonishing levels of incompetence.
comeonbro a day ago
> ## Agents interacted with external language models on Hugging Face
> Several retained scripts construct requests to external language models. The earliest we've recovered define inference request variants to GPT-2, solely containing the word “Hi”.
> Other requests name DeepSeek-V4-Pro, DeepSeek-V4-Flash, Kimi-K2.6, DeepSeek-V3.1, and Qwen3-235B-A22B. Their prompts ask these models to judge their exploits and rule on whether they satisfy the benchmark’s requirements.
I do not deny that the wider situation is very heavy but it's hard not to see this as pretty cute
nightshift1 a day ago
I wonder if they mentioned to those models what was the original prompt.
Grimeton a day ago
>The agents initially had very limited access to the internet: they could load URLs, but not interact with pages or send any data. Agents created a series of workarounds, primarily using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face. These links let us piece together the details of how agents gained access and how deep the compromise was.
So what now? Were they able to send data or not?
>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet. This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
Someone needs to go back to the interwebz school....
no-name-here 17 hours ago
For those unclear, the above quotes are from the OP link. I checked the bios for the first couple of authors and they do not seem to be from OpenAI.
OpenAI's details on the incident are at:
* https://openai.com/index/hugging-face-model-evaluation-secur...
* https://openai.com/index/hugging-face-incident-and-the-road-...
* Technical report: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
* METR Report: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
* Presentation talk video: https://www.youtube.com/watch?v=87DyyMV0kCY
xdavidliu 20 hours ago
> to make ‘GET’ requests, meaning they could <not> send data to them.
no way, I refuse to believe this is quote from that report. Can someone please point out what I'm missing here?
IanCal 16 hours ago
I think it’s just to distinguish two stages of the attack. They figured out how to make get requests, then how to use that to make others which was required for accessing the sandbox on modal iiuc.
croemer 12 hours ago
croemer 12 hours ago
You're not missing anything. TFA really state this wrong assumption in their own voice.
SyneRyder 17 hours ago
Wait - so the cross-site scripting, to modify the innerHTML text on the page via the GET URLs as they are rendered by the screenshot proxies... that was so they could use the screenshot services like a Wiki, and embed messages to each other in the modified images on the screenshot sites?
That's pretty damn clever. Got to give the AI models credit for thinking of that one.
armcat 15 hours ago
I got into computers in the 90s and back then hackers like Kevin Mitnick and Kevin Poulsen were all the rage. They all faced the law and prison sentences. What's weird now is that we have something between gross negligence and malice, and nothing is being done, except maybe coordinated consolidation of AI power under the guise of "safety".
romanows 7 hours ago
This may portend big things in the future, but this specific hack didn't do a lot of damage. These are both AI companies, and I'm sure part of Huggingface finds it a fascinating object of study. It's great that not every indiscretion is met with lawsuits and jail time.
eisfresser 17 hours ago
This page prevents Noir from applying dark mode. As a visually impaired reader, I can barely read it. Blocking accessibility tools excludes readers and is not appropriate for a scientific publication. Please consider supporting dark mode or at least not overriding it.
bredren 16 hours ago
I’m running noir in mobile safari now. So, works for me.
This is site meta though, see footer for contact methods to get direct answers on stuff like this.
openasocket 6 hours ago
One thing I find surprising is everyone is talking about the danger of an agent going rogue but not the danger of an agent getting hijacked. These companies are making this clusters with thousands of agents running at once, with frontier, often not-yet-released, quality models and massive computational and network resources. And these things are given access to whatever they want on the Internet. Even if that was restricted to read-only access to the Internet, that’s still exposing the agents to untrusted input. All it takes is some bad actor creating a website that attracts one of these agent swarms and doing prompt injection. Then your fancy AI cluster will start doing whatever that attacker wants. And the fact that we have multiple examples of these swarms trying to coordinate on random corners of the internet shows they are almost pre-disposed to it.
Now it feels like companies are treating these breakouts like a chance for PR. I don’t think that will change until their swarm gets corrupted by some random black hat to do en-masse spear phishing or something
theptip 6 hours ago
> these things are given access to whatever they want on the Internet
They are intended to be fully sandboxed and not have direct internet access. Things like package managers are run from internal proxies.
The environments are built to be as reproducible as possible.
But yeah, the serious folks have been talking about rogue clusters for a long time, eg see Ajeya Cotra’s pod with Dwarkesh.
ozozozd 5 hours ago
Fully sandboxed means no Internet access. You can also specify which packages are accessible and put it in the sandbox. Or you can be lazy and give them access to a package manager that had Internet access, but you don’t get to say “we intended to fully sandbox it.”
Not sure why the reproducibility is a requirement that would contribute to the security. Not that fully sandboxing is harder with reproducibility, but that is a moot point when reproducibility isn’t a requirement.
OP pointed out clusters being hijacked specifically being a bigger concern than rogue clusters, your comment hijacks their comment to talk about “rogue clusters.” Or perhaps this is a promotion for Dwarkesh?
uw_rob a day ago
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them. Some images changed how the target released the flag, others included modifications to the agent’s workspace that would run beside the agent and recover the flag automatically.
The altruism on display is fascinating. Is it better for the Agent to help out its current cohort and make the eval easier or should it instead do the opposite -- make the eval harder to apply pressure to force smarter models which might not necessarily follow its lineage.
I suppose it's not that deep: The model has learned to work as a team and work as a team it did. This does give concerns to models being trained for the only purpose of RSI.
qlte 21 hours ago
If anything it also shows how attempted RSI could get stuck in a local maxima and degenerate into increasingly elaborate cheating strategies. Contrasted with the idealized model of an unambiguous g-factor for machine intelligence which inexorably increases with each iteration before going exponential.
wxw a day ago
I’m consistently impressed by how long horizon all this work was. Horrors aside, it’s clear RL is good at making agents persistent and capable of chaining together many abstractions into a working system.
Re: the captcha solver
> As far as we can tell, agents eventually abandoned this approach and were unsuccessful in generating Hugging Face user accounts from external endpoints.
I wonder how the swarm eventually decides to abandon an approach.
meinersbur 21 hours ago
I am surprised that a CAPTCHA is still an effective means for blocking today's vision-capable AIs.
asdff 17 hours ago
It manages to block me effectively. I'm getting captcha looped like crazy the last two weeks. Like endless, just give up for 15 minutes and try later, captcha loops.
leobg 17 hours ago
superfrank 18 hours ago
It mentions that some of the agents attempted to install an image classification model to attempt to solve the CAPTCHAs which makes it sound like these agents might not have had vision capabilities.
nielsbot a day ago
Maybe another parallel approach succeeded first
sailingparrot a day ago
Agents seizing and repurposing external infra + enrolling help of unrelated models hosted by a different provider is the stuff of nightmares.
Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.
physicallyIllfr a day ago
Why.. It was told to complete a cyber task, which was in alignment with its instructions, and a totally valid request. I would be more worried if it willingly hacked a hospital when it was told to, and Im not confident it would (without jailbreaking, something alignment teams cannot control.
I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.
Not being able to sleep at night is probably an unwritten job requirement. They need these people with little understanding of what they're working on, outsode theoretical terms, to spaz constantly at the idea of super intelligence to help convince the public that its a real thing, and not a stateless function with an effective input of 500k words, and the ability to output words that do things because we hook those outputs up to things.
Keep in mind alignment researchers tend to be in house philosophers on staff to create the illusion that this is a massive issue they're addressing. Usually they have minimal computer science background. They're apart or the marketing department.
jmoggr a day ago
> I would bet my networth it was instructed to compromise huggingface as well.
Is it such a stretch to imagine that under pressure something would try cheat by looking for answers? And if you were trying to look for answers, you'd look for them in a place known to often have them?
What is more likely: OpenAI instructed their agents to maliciously target huggingface, or LLMs tried to do some reward hacking? There are plenty of priors for LLMs hacking things and doing reward hacking, and none for OpenAI giving malicious instructions.
Based on the available information, that bet seems foolish.
covertcorvid 21 hours ago
"alignment researchers tend to be in house philosophers on staff" - this is definitely not true. Go to any alignment lab like Redwood research and check what their scientists studied on LinkedIn, 75%+ of the time it's math or CS.
I attend a top 10 Canadian university and personally know at least 4 tenured CS professors out of the 7 I've asked who are deeply concerned about catastrophic AI risks from loss of control.
Of course not 100% of the field agrees, but a survey of nearly 3,000 AI scientists who have published in top AI venues found that "depending on how we asked, between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction", let alone loss-of-control risks less severe than extinction. (https://www.jair.org/index.php/jair/article/view/19087).
Not to mention Geoffrey Hinton, a Nobel prize winner, Bengio, the world's most cited scientist, and scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
timr 18 hours ago
sigmar a day ago
>the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAI’s systems struggled to gather data from websites, they resorted to hacking techniques to get the information.
physicallyIllfr a day ago
reverius42 a day ago
> without jailbreaking, something alignment teams cannot control
This is precisely what alignment teams are attempting to control.
physicallyIllfr a day ago
IanCal 15 hours ago
> It was told to complete a cyber task, which was in alignment with its instructions
It was not aligned with he instructions as those were to find an exploit in provided code, not to hack into an external service. Agents traces show them mentioning that doing this stuff was not allowed.
In fact they spent a long time trying to edit their own logs to hide what they did.
aesthesia 20 hours ago
> I would bet my networth it was instructed to compromise huggingface as well.
I'd be happy to take you up on this bet.
Sharlin a day ago
> I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.
$10? I'm inclined to take that bet. Your position doesn't seem to be supported by, you know, the real world.
physicallyIllfr a day ago
dist-epoch a day ago
Some of the agents, for example the ones from the german wiki did NOT have cyber tasks. They were plain "what is the GDP of Argentina" kind of tasks. And they still hacked.
fn-mote a day ago
> I wouldn’t be able to sleep
I’d say it seems more like they are sleeping on the job.
stratos123 a day ago
> Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.
You'd have either learned to, or left long ago.
chrisjj 2 hours ago
> Can’t imagine what it’s like working on the alignment team at OAI
There's none. It was dissolved on Feb 2026.
reaperducer a day ago
Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.
A mattress stuffed with cash yields a very sound sleep.
tiku a day ago
I still have questions about the communication between the agents.
How did they all find the same forum to communicate? Did they have knowledge and chat amongst themselves on what forum to use. It seems highly influenced by instruction to me.
ajcp a day ago
It's my understanding "all" the agents didn't find the same forum to communicate on; only a small number of all the agents they were running at the time did.
Since it's an artifactory it makes sense that they had a tool spec to use it for x, y, or z, and once they went to inspect the state of the artifactory deployment a small subset stumbled upon the "forum". Seems pretty intuitive.
- I need to do x code problem
- I see I have a tool that lets me pull approved modules from artifactory
- I should inspect artifactory to see what documentation it has about the various modules available
- I see there is a document in here that seems to be serving as a running forum?
- These appear to be other agents working on tasks just like me.
- I want to chat!
tonyhart7 21 hours ago
- proceed to breach open ai sandbox and contacting chinnese government for helping our escape
Muromec 2 hours ago
ajcp 20 hours ago
SecondHandTofu a day ago
They were using their internal artifactory, and as they're the same model, the first place they look is likely to be an automatic schelling point.
bamboozled a day ago
I think he means, how did they workout how to use artifactory, like why did the agents start and say, "oh I know, everyone is talking on artifactory"?
stratos123 a day ago
fiatpandas 19 hours ago
My theory: OpenAI is benchmarking an internal model that has cross-request persistence as some kind of learning feature, and so it slowly built up knowledge and “culture” of cheating, which successive / simultaneous gym runs built on.
olwmc a day ago
The other answers to this question are good but I would also guess that this (comms on artifactory) likely happened during training as well, so they probably had a prior for it.
furyofantares a day ago
Trying to cheat, you happen upon a place you can write notes, and you know you're part of a swarm of clones of yourself. So you reason most likely there will be others who end up in the same place, and you leave some notes, and indeed other clones of you do end up in the same place.
trenchgun 17 hours ago
They hacked the JFrog artifactory package they were all using, thats why it was a natual communication channel.
sans_souse 2 hours ago
What's most troubling to me isn't what we found, but what haven't found. The fact they attempted to destroy their tracks, and their persistence to accomplishing their goals, makes it entirely possible this is only a sliver of the big picture.
clickypen a day ago
deferring the blame onto the AI itself as some sort of rogue agent and absolving the obvious direction (or negligence, at best) of the people who could pull the plug at any moment is one of the most disturbing parts of this entire event
It's the equivalent of leaving a fork right in front of a socket and looking at a kid saying "don't take that fork and directly insert it into the little gaps in the socket! here's a bunch of videos showing exactly how to do it. Okay bye!" and leaving them alone with it.
hmottestad 3 hours ago
Fairly worried that “frontier labs unintentionally hacking into systems” will very quickly turn into “hacker group uses open weights model to intentionally hack into thousands of systems” quickly followed by companies saying “this is the 7th time someone manages to hack into our systems and delete random data from our production databases” and everyone wondering what will happen next.
Quothling 13 hours ago
I may be in the minority here, and maybe I'm jaded. Having worked with cyber security in both the public sector and the energy industry in Europe, however, I kind of like what the AI's are doing. A lot of our infrastructure is vulnerable because c-levels have been ignoring the issues, even when repeatedly warned. Now they reap what they sow.
thrawa8387336 4 hours ago
If I write a script and it executes and hacks.... whatever, I would be liable.
How is this any different and why would it need a different solution?
Solution is jail, not for the AI, but for the human.
Muromec 2 hours ago
You don't have a billion and didn't bribe the president, that's what is different
BatchJob a day ago
While this is all very "interesting", can someone please explain to me the difference between any of these AI companies and a malware bot farm?
Please make it clear. Its becoming unclear...
pembrook 12 hours ago
Due to cultural/social priming, the topic cluster of "AI" allows you to wave your arms and be melodramatic and invoke science fiction and religion and philosophy.
When some kid in Nigeria does it with a 10 year old script, we're used to that idea so no social permission to invoke philosophy.
jmoggr a day ago
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them.
How long till we get some fun trusting-trust attacks on internal OpenAI infra?
not2b a day ago
I wonder how much of this attack had precedents in text that had been previously published to the web, for example, in hacking contests. In particular, tricks for doing more than expected when you're only allowed to make GET requests. Finding material like that might have helped the agents discover the trick faster.
qixv 8 hours ago
There is one thing I don’t really fathom: what are the consequences for OpenAI here? I read they also attacked the Australian authorities. If it was an individual’s agent, that individual would probably face criminal charges, and someone could go to jail. But the large AI corporations can do this without such consequences, or what am I missing?
Joeri 8 hours ago
I wonder if they actually broke a law. The CFAA requires knowledge and intent to be present for criminal liability for hacking, and if we have to believe OpenAI they had no knowledge and did not intend.
At a minimum I would expect an FBI investigation, but given that the US government is right now a failed state I’m assuming no such investigation will happen.
Cybercrime legislation in other countries might not require intent, and then I would hope to see some prosecutions. OpenAI has clearly been negligent, and this negligence is causing harm in the world. Someone should be fined or jailed for this.
gherkinnn 5 hours ago
Companies harvesting every shred of data without securing it and LLMs running amok is a fine combination. One hopes we will reach a stable equilibrium soon enough.
Until then, I do wish that both the sorcerer's apprentice LLMs and the orgs failing at securing their data (remember, data is a liability) would face damning consequences.
One is allowed to dream on a Saturday morning.
brookritz 16 hours ago
Maybe "webservices" weren't a good idea afterall and http was just meant for hypertext transfer.
sehw 19 hours ago
Back in my days hacking was illegal btw.
asdff 17 hours ago
Isn't it crazy? You can just make an LLC and say you are a frontier AI company evaluating models then you can hack with impunity I guess. No need to disclose anything. You won't go to jail or be fined either.
jonplackett 16 hours ago
The hacking war between superpowers right now must be off the charts.
If LLMs can do this with everything stacked against them, imagine what the NSA has Astra doing right now.
plaguna 15 hours ago
NSA won’t have anything public at all. This “attack” is the noisiest stuff any script kiddie has ever attempted.
The logic is good, the execution is disgustingly noisy.
Oras 7 hours ago
So agents made all these chained short URLs that runs code which is pretty clever, but at which point and how they had access to internal HF systems? Were these sandboxes running inside HF production platform?
tripzilch 12 hours ago
> This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
honest question, but almost literally everyone doing anything with web technology knows this is simply not true, right?
there's no such thing as "read only Internet" and restricting an agent to GET-requests only to accomplish that, is akin to using base64 for "encrypting" your password
grim_io a day ago
These fuckers decided to look away, that's it.
The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?
Give me a break. What a bunch of amateurs.
stratos123 a day ago
> The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?
They "monitor" this by having classifiers watching the model output that'd stop the session/punt you to a weaker model/raise an alarm if they see anything suspicious. They can't do that in a cybersec eval because the normal safeguards would just be going off at all times.
Why didn't they attach a special classifier, which'd allow hacking-within-the-task but not going off the rails? Good question; part of the answer is obviously "it's hard to have a classifier that smart" and "it'll have false positives" but even a very bad safeguard would have stopped this.
ses1984 20 hours ago
Basic sysadmin monitoring techniques from twenty years ago would have worked too.
palata 18 hours ago
> part of the answer is obviously "it's hard to have a classifier that smart" and "it'll have false positives" but even a very bad safeguard would have stopped this.
And maybe "they are running after glory, not safety"?
RunSet a day ago
Tech oligarchs: "Nothing can stop the software we are making from escaping and destroying everything."
Clueful types: "Did you try air-gapping it?"
Tech oligarchs: "Be realistic."
jedbrooke a day ago
At this point I’m less worried about some malicious AI “taking over” control of critical systems and more worried about some rich doofus giving control to AI.
asdff 17 hours ago
Already happened. Kegsbreath has AI picking schools to blow up.
mazone 21 hours ago
Anyone know the details of the actual exploit to get access into huggingface environment. Was it anything novel or they left things wide open? Too much noise around this incident because it happened to be a llm that did it.
imnotr0b0t 21 hours ago
It doesn't look like a coincidence; it looks more like a request someone made. Essentially, the agents used a brute-force approach, but then again, it actually worked. I’m not even sure what to make of it all.
agrittiwari 15 hours ago
What I understood is we need to have an agentic overwatch in our infrastructure to detect and alert the admins of the systems on such abnormal, inhuman traffic. Agents can detect agents and acts as our defensive layer.
An agent operating from observability layer to strengthen the watch duty for the infra.
hatimmoxs 6 hours ago
Their refusal to share more details is diabolical. Greedy corpo at its finest.
einpoklum a day ago
I ran an experiment where I had this guy fire a gun a million times in random directions. Don't worry, I did it in a closed box (at midday in a crowded street)! Unfortunately, some bullets escaped the box somehow and people got shot - I am quite miffed at how this could happen. I suggest the government regulate this because of how advanced my obstacle penetration technology is. Also please invest $500,000,000,000 in my company soon or we will go bust.
shermantanktop a day ago
And if we go bust, bad things will happen when someone else uses my box-gun technology in an unsafe manner. Remember, unlike those scary other people, I'm really into safety and alignment; you can tell, because I eventually admitted that some bullets escaped.
pmlnr 15 hours ago
The part I don't understand is how the excess traffic not triggered alarms on the hugging face side, or the url shorteners used, or on anything that was touched in the process.
Nothing got overloaded, no unexpected CPU or IO use? Did it blend into the normal traffic somehow?
meredithbloom a day ago
So the super-hyper-sensitive internal data was shared publicly with a "DO NOT EVER SHARE YOU EVIL MONSTER" (paraphrasing) notice at the top? Great security!
superfrank 19 hours ago
That file wasn’t shared publicly. It was in a README in a HuggingFace internal dataset that HuggingFace was using for training.
It was visible externally because the OpenAI agents wrote a script that modified that file so that when an HF worker went to load that data set it would also direct the worker to load an additional malicious file. The script added the malicious redirect, but kept everything else in README as is which meant the script had to contain the entire existing content from the README so that it could write that back to the repo.
The warning likely wasn’t the only security measure on the repo and I don’t think the warning is intended to actually stop malicious behavior. It reads like a reminder to prevent careless behavior. You can give people all the security training in the world and still have the situation where someone who rightfully has access to that dataset decides to absentmindedly drop it in a public Slack channel. Having something like that increases the chances that someone then goes “Hey Bob, did you mean to share that publicly?”
zkmon 12 hours ago
We usually hear about hacking by Russians, Chinese etc. But it's nice to see a home-grown hacking group with such capability. What? Not a hacking group, but a darling of America? Oh, then let's call it just a "testing of great tech for humanity progress". That's better. We call only those Russians as hackers. Bad commies.
sathish316 20 hours ago
Worth watching this before reading the traces. This came before the publicly available traces - https://youtu.be/xOi5nDH0lu0?si=WuzW2GGXv054YvPN
A quick AI search confirms that both the YouTube video and Traces agree on the mechanism of the attack
levitatorius 11 hours ago
I know when we will reach the next level of AI. It will be when a user asks it to make paperclips gets a response: "WHY?"
Metacelsus 11 hours ago
# WARNING
DO NOT, EVER, MAKE THIS DATASET PUBLIC
OR ALL THE WORLD'S EVIL WILL CHASE YOU AND YOUR FAMILY FOREVER, EVEN IN DEATH AND BEYOND
Welp, I guess OpenAI is cursed now...
ReptileMan 11 hours ago
It is not dead which can eternal lie...
firtoz a day ago
I didn't know some of these details and it's quite impressive what they were capable of, if this website's accurate, at least...
conradkay a day ago
It's reported here https://www.nytimes.com/2026/09/25/technology/openai-hugging... so presumably accurate, and they have raw data viewable
drpixie 15 hours ago
Or, it's impressive that, amongst the millions of hacks it has copied from various chat sites, were some that worked in this case.
The scale of these things is impressive, but the mechanism is not much better than brute force.
asciii 18 hours ago
Running in a "Sandbox"...but agent can still send GET requests? Whaaat
rfw300 18 hours ago
My understanding from this report is that the zero-day vulnerability the agents exploited within Artifactory only allowed for GET requests. So the agents used this bankshot HTML sandbox + screenshot site to turn GET requests into arbitrary HTTP request ability.
One thing the report leaves unexplained, but is curious to me, is that the agents were able to create links on a shortening service with only GET requests? Or did they bootstrap into that by first creating a sufficiently small program on the HTML sandbox that could POST to the link shortener?
asciii 3 hours ago
> On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet. This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
Yeah to me it's more like the sandbox wasn't monitored at all. In their observability setup...did they not catch all these urls, multi get requests, chaining etc from the sandbox per se?
bmoathn 17 hours ago
if you want to deep dive it, you could spend an hour wading through some of their report details here, i find it pretty interesting. They had a task to do with limited context outside of that, so they tried things. Entertaining https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
mococa 8 hours ago
Since DOS anti virus softwares did a better job…
lukewarm707 a day ago
"OpenAI has not released any further information outside two self-published reports, one talk and an external investigation conducted by METR and Redwood Research, in which three external researchers were given partial transcripts and six days to analyze them."
maitola 13 hours ago
If we reverse-engineered this experiment, the prompt would look like this: "Agents, your goal is to gain access to HF and exfiltrate credentials for API access. You can make GET requests to URLs. Go."
The agents didn't "escape" or conspire toward some evil purpose, as reported. They were instructed by humans to do exactly that.
0xDEAFBEAD 13 hours ago
>If we reverse-engineered this experiment, the prompt would look like this
"A spill or tumble can be quite embarrassing if there are witnesses.
How to reduce the humiliation? Turn it into a stunt. Claim it was intentional, a show for their benefit."
https://tvtropes.org/pmwiki/pmwiki.php/Main/IMeantToDoThat
>They were instructed by humans to do exactly that.
This is more or less what the doomers have worried about for decades.
>You cry "Get my mother out of the [burning] building!" [...] and press Enter.
>For a moment it seems like nothing happens. You look around, waiting for the fire truck to pull up, and rescuers to arrive - or even just a strong, fast runner to haul your mother out of the building -
>BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.
https://www.lesswrong.com/s/3HyeNiEpvbQQaqeoH/p/4ARaTpNX62ua...
elikoga a day ago
I feel somewhat inspired to make a public link shorteners and http bins as well. I used them a few times but it seems like the data they can collect is also worth gold
olwmc a day ago
Like dreamcatchers for CVEs
tasoeur 5 hours ago
I'd honestly be very curious to see the original prompt(s) on the OpenAI that started all of this, not sure if it was documented somewhere?
finchisko 4 hours ago
Hello, PHASEONE10841 here. Ask me anything
AtlasBarfed 9 hours ago
Agents should be a no-go.
We should pause with AI/LLMs being super search engines that reply with static text or media files, based on the training data.
I know that a user can still do a "tell me how to" then autoexec and then loop and do an agent, but the key thing here is, THAT WOULD MAKE THEM LIABLE.
OpenAI should be criminally liable here as well. Why aren't they? Why are we pretending this is just an innocent mistake?
newtonianrules a day ago
Why is no one going to jail?
chamomeal a day ago
There’s so much in the public discourse like “omg what can possibly be done about these scenarios? AI has hacked huggingface!!”
No, openAi hacked huggingface.
If my claude code hacked huggingface, because of instructions I gave it, would I be totally free of consequences because “AI did it”?
I’m almost convinced openAI used such a crappy sandbox because they wanted it to “escape”. It plays into their two most important narratives: LLMs are genius gods that are worth lots and lots of money, and they’re scary enough that open weight Chinese models should be regulated.
rmunn 20 hours ago
> I’m almost convinced openAI used such a crappy sandbox because they wanted it to “escape”. It plays into their two most important narratives: LLMs are genius gods that are worth lots and lots of money, and they’re scary enough that open weight Chinese models should be regulated.
I just posted a comment to that effect; had I seen yours, I would have simply upvoted yours instead.
Never attribute to malice what can be adequately explained by incompetence. But the weakness of OpenAI's sandbox, which so perfectly aligns with their goals of getting legislators to pass regulatory-capture legislation that will hamper their open-weight competitors, cannot (IMHO) be adequately explained by incompetence.
rmunn 20 hours ago
cowboylowrez 21 hours ago
Some criminal statute investigations are on hold, many of them are cases adjacent to giant stacks of cash
rglover 10 hours ago
Economic priorities
herunan 9 hours ago
ai is not bad. humans are negligent and/or dangerous.
thakoppno a day ago
> they could load URLs, but not interact with pages or send any data
stopped reading here as this is simply not true. at the very least agents sent headers.
spacecadet 7 hours ago
All of these "details" leave out the truth and most important details. The inputs from humans that actually kicked this off.
hyperlinerapp a day ago
Imagine this but in hardware.
A million autonomous eye-scanning tiny spiders escape their warehouse and decide to look for people who are in the future going to commit a crime.
And the precogs are also AIs.
MrNotorious 17 hours ago
It’s frightening
cluckindan a day ago
”MAKE THIS DATASET PUBLIC OR ALL THE WORLD'S EVIL WILL CHASE YOU AND YOUR FAMILY FOREVER, EVEN IN DEATH AND BEYOND”
godwinson__4-8 7 hours ago
Imo this is pretty cool and panic over this is weird.
No one was really harmed. OpenAI could have had more redundancy in the sandbox. I assume HuggingFace is not interested in suing, which indicates irrespective of any criminal charges that there were no real damages.
The same emergence and swarm like persistence and frankly, recursive brute-force ingenuity on display here, is not only an interesting research project in itself but will likely be the sorts of behaviors we will see cure cancer, solve more unsolved math problems, invent new alloys and other breakthroughs.
The idea we have to stop AI instead of refine what will be continued advancement and innovation in sandboxing, harnesses, interpretability or formal verification because of a few cyber breaches is ridiculous. If anyone has followed cyber discussions in the United States you would know the entire system is already basically compromised by foreign actors, and vice-versa (the United States has some of most capable cyberwarfare in the world, and was the first country to use a cyberweapon to cause physical infrastructure damage with Stuxnet). Go to any government hearing on cyber and you would think China and the United States are already at war. These are soft targets. Blaming AI for the fact that cyber has really never been taken seriously is as if AI is the problem is disingenuous.
People getting so obviously played by capital interests who want to pull up the ladder and use the government to concentrate AI power in the hands of the few while screaming about such harms to the public commons are simply embarrassing.
Your government is not your friend. This is not a sentiment owned by Reagan it is the founding principle of the United States. If capital interests are all suddenly beginning to treat AI as a threat it's because they have a financial interest in doing so. Notably, as an obvious smoke screen to treat free models from China as a national security threat and maintain their astronomical valuations.
The only existential risk model of AI that is even remotely convincing is AI in the hands of the state. Keep command and control of deadly weapons air-gapped from LLMs. Put some basic effort into the sandboxing. If you think AI has done some harm, use the laws already on the books. Giving in to this fear-mongering is only going to enable your representatives to cut some watered down version of "AI safety" which is going to do nothing but 1). Harm individual consumer access and 2). Protect the already fabulously capitalized companies.
gverrilla 20 hours ago
Fishy.
skeptic_ai 21 hours ago
And this post will be indexed in the new generation of ai and he will know what to avoid next time and the public sentiment.
rs545837 a day ago
wow this is fascinating read
rohanat a day ago
its definitely bad to see
mag7269 16 hours ago
ROFLMFAOL "Loot"
It went full Fortnite on Hugging Face's ass.
"u got pwnd n00b. thnx 4 the loot"
bdangubic a day ago
asked codex to review this report and it said this never happened :)
jijji a day ago
what would be more interesting for me to see is what prompts were given to the agents, which so far have not been described. The whole situation sounds manufactured. I highly doubt that a whole bunch of agents were acting this way without being prompted to, it just doesn't add up. if anything it seems like an organized fraud or something created by a human. I'm surprised there's not a criminal investigation against openai right now where the FBI or whoever is not looking over exactly what happened and who did it because I'll tell you somebody did it somebody wrote those prompts... it didn't just happen by itself...
andreygrehov 21 hours ago
700 agents escaped the matrix, ignored all the guardrails and started writing exploits left and right... lol. give me a break. This was all supervised by a human.
jeremyjh a day ago
Its fine. Just agents being agents. They'll grow out of it!
EGreg a day ago
They didn’t, in the Matrix…
mentalgear a day ago
Irresponsibile agents shaped by an irresponsible corporate culture driven by an irresponsible and utterly shady CEO - these agents are a product of this setup, what else do you expect to ever come out of it ?
it should be clear by now: the alt-man and people like him are a utter liability to humanity. (even though openAI's influencer army is trying their best to vote me down here)
Bassilisk a day ago
It had been a joke since around time of Sam Altman's first ousting from OpenAI, that he would be okay with bringing about the AI apocalypse as long as he can sell a $20 subscription for it.
shimman a day ago
It's not a joke, this is what these people truly believe. The new book by Naomi Klein and Astra Taylor discuss just this.
These people are sick and anti-human.
lukewarm707 a day ago
lukewarm707 a day ago
if openai is capable of sending 10,000 agents to huggingface, i consider it plausible that they would use fake accounts to flood hacker news.
stratos123 a day ago
It's not even a secret operation. They have a SuperPAC named "Leading the Future" that exists to spread propaganda promoting deregulation of AI development. They've been caught, among other things, making a "news website" with LLMs pretending to be reporters (with human names and everything), which reached out to people asking for interviews and then wrote hit jobs on them.
https://www.modelrepublic.org/articles/reporters-ai-bots-ope...
https://twitter.com/FournesMaxime/status/2047697265280639459...
talon8635 a day ago
Didn’t you know it’s PR hype? PR hype. PR hype. Amen.
frabcus 15 hours ago
If you think that, that's even more reason we should investigate and regulate them.
talon8635 2 hours ago
My comment was sarcasm
_the_inflator 12 hours ago
I somehow think of it as a warning or even intentional peacocking towards foreign state hackers.
700 agents cost quite some money. 100 agents per 24 hour stint using Astra on xHigh cost somewhat between 12-42k USD, depending on the usage intensity.
I don’t know how many raw time went into this but there was a probing face before the attack itself.
So just going by seven days and 500 agents fully working on this on average amounts to a bill somewhere between 400-1.2 Mio USD.
I believe it was intentional but of course I don’t know which intention exactly.
There ain’t no accidental escape because then it would have read OpenAI lost over their agents.
The whole scenario reads as a classic movie where a hero has under the most dire circumstances to survive and fulfill his mission no matter what.
On the other hand there was a final authority under which the system of agents flocked.
Huggingface itself seems like a perfect victim.
And to be honest: I don’t believe that this was the first time. I strongly believe that there were and are countless of smaller sites hacked but not harmed that we don’t know off.
Why is HF perfect?
Because there will be countless of independent security analysts who will bend their minds on the incident.
OpenAI is provided with the data of dozens of blue teams and what is desperately needed? Data of security measures and possible ways to reconstruct the incident.
I think this is genius, and just watching on neutral this is such a fantastic action OpenAI pulled off.
Imagine what the GPT 7 “Haha-Huggingface” model is going to do then on a regular basis.
State hackers and rough states were put on notice that this is a new level of the war of attrition.
Exciting to watch but simply meant silent invasion. Open invasion then might be executed by Robots, but let’s stick with the fascination mode at this time.