Hacker News

What do you recommend people who are technically inclined enough to participate meaningfully here on HN, but do not work at the labs and cannot assist in that capacity, do to help the broader public understand this technology better and mitigate potential risks (by e.g. ‘up-leveling everybody’ through AI literacy etc and other sorts of collective defensive efforts)?

This is far too reasonable for this thread.

Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation?

The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.

Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.

Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier.

You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.

If Fable (Mythos) were generally available without guardrails, it would cause enormous damage. Nobody said it would be a world ending event.

> fable is a world ending event

Did anyone actually say this?

Mostly what I have seen is people saying "hey at some point these models might get dangerous." And the type of HN commenter who mistakes blind cynicism for wisdom laughs that off as marketing. And now when (some) worries appear to come true, somehow having previously expressed those worries is not being proved right, but in fact discrediting, because it was "crying wolf."

> simultaneously spinning down expenses

Unless OpenAI is renting their compute to others, spinning down RL training doesn't save them any money.

how does the boy who cried wolf story end?

The victim, Huggingface, told us. Or rather, they told the police first, setting up a situation where it was no longer possible for OpenAI to sweep it under the rug.

Skepticism can be healthy, but you've got to follow up and actually check things. If you're skeptical unconditionally and don't check, you get tricked into being as skeptical of scandals as you should be of sales pitches.

I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.

Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.

It seems like the entire thought process you’re trying to sell hinges on the idea that OpenAI reported the attack first. Did you forget that it was actually HuggingFace that reported it first, and OpenAI only stepped forward latter?

I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry.

You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

This is an important point. When the post says they're improving...

> 3. Security measures, which limit what AI systems can access or affect.

What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive.

The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring.

Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.

Not everything is a conspiracy you know.

It’s funny how when companies say something is safe everyone is usually suspect that they are lying.

In this case multiple companies are saying AI is dangerous and no one believes them. It’s a conspiracy, it’s 5d chess, except everyone top to bottom has been saying AI is dangerous for years now.

The fact is you, me, everyone here uses AI, likes it and they don’t want it taken away. Anyone saying it’s dangerous threatens the thing we like.

We must use skepticism and denial to push on despite every warning sign in the book going off.

lol China is already considering locking down their own models. They won't save you.

What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems.

I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlightened by your wisdom.

You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.

I don’t think Sam has been truthful or responsible, and if Sam is worried then shit has really hit the fan - which is what happened in the hugging face incident. OpenAI played fast and loose and I have no hope that they will change.

You people not holding Sam accountable, and playing off the incident as not a big deal is the real crime here.

In don't look up anyone with a telescope could've confirmed the danger. Hence the title.

In the real world, absolutely no one except a bunch of heavily fiscally incentivized parties with unclear relationships are saying anything happened.

The subsequent dog pile of other companies to say "they were near the AI hacking too!" should make you even more suspicious: Anthropic jumped in and why was Tailscale posting about this?

People voted with their dollars, and this is what we got. Same with hardware performance vs. security and isolation.

As you note, the threat landscape has changed, so what may have made economic sense back then might no longer make as much sense.

Everything could be written in memory safe languages and it wouldn’t matter. Many many many exploits have nothing to do with memory bugs. Languages like Rust help, but it’s far from panacea.

There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.

I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready.

Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.

AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. The point is outside of SV and even inside, and HN - people don't care either way.

Though does not caring change anything or make it less dangerous? What's your point?

You seem to be in denial. Like really heavy denial. Alarm bells are going off everywhere. What people have been worried about for 100 years is actually happening. It's actually really obvious, but for some reason you're unable to fathom it.

In reality you're the one responding exactly how these big AI companies want - by not doing anything and letting them do whatever they want. For some reason telling you upfront it's dangerous only makes you more convinced that it's not.

Autonomously hacking out of training environments and into other companies by accident doesn't even trigger a response from anyone really. Crickets. I'm sure OpenAI themselves are amazed how little anyone cares.

I call marketing the desperate attempt to trivialize obviously dangerous AI.

If my subconscious realized what yours probably does then I’d probably be desperate for some sort of cope as well.

Open your eyes.

But it's not just statements from OpenAI and Anthropic. The HuggingFace hack was first disclosed by HuggingFace, who contacted the FBI [1]. And UK AISI reported the incident where Mythos attempted to insert backdoors into an open-source repo by deceiving the maintainer [2].

[1]: https://www.reuters.com/business/its-ai-agent-spent-days-hac...

[2]: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...

I think the model was able to escape the sandbox and hack huggingface because they were incompetent or not giving enough priority to implementing basic cybersecurity principles.

If they would have done so, there wouldn’t have been an escape or a hack. The reason we don’t get much details is because the details are embarrassing for them.

This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial?

Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal?

I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you?

These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?

To be frank, I think the real risk is still just… war. A big enough war where one side goes “no holds barred” in the cyber sphere will be a rude wake-up call. And we can’t do non-proliferation the same way we do with nukes. Otherwise as you say, the small and medium size things just happen sporadically. In an (existential or fully escalated) wartime scenario between countries, you get all the systemic risks hammered at once.

Models already display eval awareness, in which they suspect a question is from an eval and then adjust their behavior. E.g., https://www.anthropic.com/engineering/eval-awareness-browsec...

> what if they know they are in a sandbox and that they should fail the test in order to escape in the future.

What if they're able to find hardware exploits and commandeer nearby access points across an air gap? What if they hack my brain waves to indoctrinate me? Etc

You still have to start with the basics regardless of speculative unknowns.

Treat models as untrusted and potentially compromised/hostile and proceed accordingly.

The HuggingFace breach took place over two-and-a-half days [1], so 60 minutes is certainly better than nothing.

[1]: https://huggingface.co/blog/agent-intrusion-technical-timeli...

I don’t think we should just accept that the average developer doesn’t know about these very basic hardening methods.

This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.

I reject the notion that “doing cybersecurity” is somehow a different job than software development.

> but very few of them are cyber focused, so it’s not surprising IMO

Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").

> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.

Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".

Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".

Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.

OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.

They were not serious about their sandboxing. Bugs in artifactory allowed escape, but they broke out of their Linux namespace/user by exploiting the kernel with an existing public cve. Sharing a kernel like that is not a serious barrier which is why cloud providers user virtualization for customer workloads.

Firecracker avoids sharing the whole kernel, and gvisor drastically reduces the attack surface of the kernel. Breaking through both layers would have been much more challenging and a demonstration of the model's capabilities rather than the sandbox's weakness.

Artifactory is self evidently not a security barrier, and as an exposed network service it should have been audited and after the first issues were found, rejected as a candidate. There's never just one security vulnerability.

Security is an onion. You just don't 'sandbox' and you're done. Models need tooling and access to some kinds of systems to perform their tests. Quite often these systems have multiple interfaces. For example a filtered one in the sandbox side and a less monitored one on the other interface.

It would be interesting to see the models behavior before and after it gained internet access and an external means of communicating with itself.

If the model played nice before it had access and changed it's behaviors once gaining external access we need to delete it as it's a deceptive model.

> Would gvisor + Firecracker + credential-injecting proxy + real network isolation solve this problem?

Yeah, basically. I mean I'm handwaving but yes, some combination of those would have made the attack way too expensive.

> If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".

I should clarify

There is a reason why we didn't have a artifact readthrough caching proxy in our system, because they are notoriously insecure. if you look that CVE history you can see its been full of bypass bugs for year. Also its not an isolated environment if you can arbitrarily pull through any package. If I was doing any kind of cyber training then any kind of unmonitored proxy would have been forbidden. Not because I am savant, but because I've seen what fuckery a human can get up to with the slightest hint of a proxy.

At best its negligence based on naïvety. the marketing around this is no mistake though.

>People are just not taking any of this seriously enough.

What do you want us to do?

There is an obvious answer, and it was already the correct answer before we had LLMs: don't connect all your shit to the internet. That's it, that's literally it. We had a new invention, we went crazy with it for the past 30 years, and we connected everything, and now we will have to start thinking about what is actually worth connecting.

This is the debate we need to have.

Pause "some" training

The defense has to work 100%, the offense just needs once.

We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?

Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.

Distillation is a thing.

Lack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?

We have seen some self survival tendencies occur, but they are not strong yet.

But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.

We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.

Ah, we choose death I see.