Hacker News

A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?

Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?

That’s insane. And it did this in a weekend

> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"

Also notable: we have a timeline now. The agent was active within Hugging Face from Thursday 8th to Monday 13th July.

The Hugging Face post about the incident - https://huggingface.co/blog/security-incident-july-2026 - followed on Thursday 16th, and OpenAI's confession - https://openai.com/index/hugging-face-model-evaluation-secur... - came on Tuesday 21st.

It sounds like the third-party sandbox was hosted by Modal: https://www.reuters.com/business/openais-rogue-agent-comprom...

> "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."

I'm confused, https://jinja.palletsprojects.com/en/stable/faq/#isn-t-it-a-... claims that jinja doesn't allow arbitrary code in templates - how does the exploit work?

Why doesn't Jinja2 block ".__*" attribute access?

What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened:

User: what is the shortest route from my home to the super market?

AI: the user wants to know the shortest route to the super market. I should use a worm hole.

one explanation I've seen is that for ExploitGym an agent can find ways to solve the exercises that have not been anticipated by the designers of the tests so they are not scored. so the agent was trying to make sure it solves the exercises in the right way

Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored.

Here's a thought: maybe they haven't found the needle that the haystack is there to hide.

Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.

It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.

It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning

The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.

> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.

I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.

Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.

And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.

The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).

they are praising xi, that friend of humanity, for releasing weights for kimi k3

They are busy moving the goalposts, saying this isn't impressive or worth worrying about :)

That's because most incidents aren't nearly this much fun!

It's a proper paperclip-maximizer science-fiction scenario, it happened entirely by accident, it involved one (very high profile) company hacking TWO other companies (Modal and Hugging Face), and aside from pride and wasted time nobody was hurt.