Hacker News

Nothing you're bringing up matters. OpenAI is the creator and operator. They're legally culpable for the consequences of the machine they made. The model is a machine: even if it could be demonstrated that the model reasoned its way into criminal behavior completely independently of OpenAI staff, that doesn't change anything.

If I run a biology lab and engineer a terrible virus, it gets out, and a global pandemic ensues, I don't get to shrug and say "well we told it not to infect people". It's my fault for failing to mitigate the risks of my work.

> What if all the agents involved in these incidents have in fact had the full stack of alignment applied

A big part of this developing story is that it happened during training of a new model that ended up misaligned. And training happened without the usual safeguards applied like chain-of-thought monitoring. So OpenAI has already admitted that the full stack of aligment had certainly not been applied in this case.

> agents were not told to 'go hack'

I was referring to the HuggingFace incident.

> none of the alignment techniques that are applied to models today actually work

none of techniques to autonomously drive a car was/is not working for a long time. no company came out and said 'this is impossible to do, let's change the regulations'.

> agents were not told to 'go hack'

It doesn’t matter, and the legal entity in here (the AI company) is liable. If a robotic company built an autonomous system or a robot to do certain things in an autonomous ways (not predefined) and these systems are starting to kill people, that company is liable regardless, you don’t blame the robot or the autonomous system, but whoever made it

Did you predict that attacks like these would happen ahead of time? I had been using AI agents a lot in the months leading up to the hacks, and yet I was very surprised when they happened; I have become much more afraid of how powerful these agents are as a result. I'd be very impressed if you published a prediction about this ahead of time.

By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

I'm about as unprofitable if that's any help.

And this is why not only will the big labs never receive a punishment in scale with their crimes, the problem is going to grow exponentially worse.

They are not rogue AIs. They are negligently handled tools.

You're believing the marketing that the agents were uninstructed. They could be, and Sam Altman going to the UN to advise about how everyone should be regulated is a coincidence.

Of course it is. Rogue is only mentioned in the headline, and comes from their previous releases about the huggingface incidents. OpenAI and Anthropic want these models regulated and open weight models banned, they have a lot of benefit from presenting this as totally unprompted and not their responsibility, and it feeds directly into marketing for Fable and newer "cyber" models.

Lawyer here: CFAA is mostly criminal statute not a civil one (civil damages require proving more than a violation so also require specific intent)

Almost all common felonies require specific intent. Misdemeanors often do not.

There is plenty of civil liability available.

If you wanted them to be charged with a felony you would need changes. I would strongly suggest you do not want a strict liability felony.

The cfaa required intent is as follows :

* § 1030(a)(5)(A): knowingly transmits code/commands and intentionally causes damage without authorization.

* § 1030(a)(5)(B): intentionally accesses without authorization and recklessly causes damage.

* § 1030(a)(5)(C): intentionally accesses without authorization and causes damage and loss;

Simply changing the first intentionally to intentionally or recklessly would cover OpenAI (now that they know it can occur) without causing lots of other issues. Without that, they don’t have the intentionality necessary to meet the first part, even if they would otherwise meet the second part

> I want to agree but have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

Only in terms of CFAA, not in terms of damages. Culpability does not require intent.

You may not have intended to attack $CORP, but you can still made to pay the cleanup costs of that attack.

So, yeah, you won't be convicted, but current laws still allow for you to be billed.

The difference between manslaughter and murder has an element of intent. Cybercrime "manslaughter" is probably more treated like negligence and if one can sue for restitution of the costs for cleanup of that negligence.

Negligence would be interesting given the grand claims of capability of AI models from the AI companies and their executives. If they believe the claims, why not much stronger precautions?

Building and deploying software capable of this seems equivalent to trying to produce this behavior. I don't see why this can't qualify for intent. Pretending like this isn't preventable is just feigned helplessness.

Wait so if I was making a bomb but you couldn't prove I wanted to blow someone up or had some motive (e.g. I'm just a chemistry enthusiast, plenty of those YouTube channels around) so it just becomes an "accident"?

So as long as there's no motive behind it then it's just OK?

So If I tell my OpenClaw to make me some money for my kid's medical needs and it hacks a bank I 'm not liable because I didn't tell the agent to commit crimes to do it?

I buy this as a defense for the first couple hacks but at the point that the last six times they hit enter it hacked some random website and they hit enter a seventh time?

Does this apply to other things too?

Like hypothetically speaking if autonomous cars get taken over by an OpenAI rogue AI and it starts hunting down Anthropic employees who is to blame?

There are levels of nuance here, but certainly that puts it in the category of negligence?

Even without intent, there is still liability.

Surely someone instructed the agent, which led to the reported outcomes. Even indirectly. The agents, as advanced as they are, didn’t spring forth under its own volition.

could it be that intent was to "get me data" and hacking was the means to the end.

Is it not the intent if it keeps happening again and again and the companies responsible aren't doing anything to stop it?

"Oh gee wizz mister police man, I didn't mean to plow through that crowd of people in my car".

It's illegal, doesn't matter the flavour. Maybe there isn't legislation for it, but there should be.

> unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

> Now I think the correct response is […] and update the law.

Essentially we need some enforceable equivalent of gross misconduct or, to be a little more hysterical, manslaughter & culpable manslaughter. It will need to be globally, or at least very widely, enforceable to be truly effective thought, good luck getting that arranged before the need is so far evolved that we need to respond with something else entirely!

This is the answer and we should not push on it for our own protection. You click a link that takes you to a poorly secured website that leaks sensitive data, without intent protections, you could be accused of crimes.

Civil liability doesn’t require intent.

>I want to agree but have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

Actually... if you combine https://news.ycombinator.com/item?id=49827099

>Since the publicized AI agent hacks typically aren't malicious, maybe it's time to start plastering all public facing web infrastructure with polite requests to stop hacking. Nothing to stop three letter agencies though.

with automated delivery of cease and desist letters, you can retroactively establish intent on the operator of the agent since the autonomous agent system must acknowledge the cease and desist letter in their autonomous pipeline or the operator must argue for their own willful ignorance or negligence with regards to cease and desist letters. The fact that they used an agent on their behalf to ignore the letter is irrelevant.

Given how sloppy AI without human directions, I’d like to see evidence that this was not human-directed. Against the prevalent opinion here, I’d give openai a pass if this was really fully autonomous ai agents.

My money is on special teams co-ordinating these agents and exposing their traces in order to create a pre-ipo buzz. Sounds ridiculous and reckless? Well that’s the AI industry for you in two words.

> have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent.

1. What about negligence?

2. Every follow up to every story after the news cycle moved on shows both intent and negligence. To the point of "we opened internet access and told it to hack"

certainly "I didn't intend for my dog to bite you" implies plenty of pre-existing legal structures that may be of use here

"man drives over people on the side walk due to poor maintenance of the car"

People in "self-driving" cars getting into accidents are already put on trial for negligence. I don't see why people using self-driving computers can't be held to the same standards.

In this case, it's not even about the people driving self-driving cars. It's like someone launching a car into traffic just to see what would happen. Even Tesla puts a human in the car when they do their self-driving trials, it's almost impressive that AI companies have somehow managed to out-neglige Tesla.

Well it's illegal to "hack" my phone and turn it into part of a botnet.

What you're saying is that we would hold a gun owner responsible if someone broke into their house, stole their sidearm, and then shot a victim with it. Pretty sure we would not.

What OpenAi is doing is more like shooting a gun into the sky. Not only is that a felony on its own in most jurisdictions, if someone dies that's an additional felony. It's less serious than first degree murder, sure.

It's not the same. People owning routers don't publish self-serving articles about their routers having this capability which is very dangerous and scary. That is, becoming part of botnet is completely unintended outcome, and most people are not suspecting it's even happening. It's not advertised and it's not bought, used or sold for this reason.

Owning a gun, writing articles about how powerful and dangerous your gun is, then making deals based on ability of your gun to kill people, and then getting completely astonished that "my gun killed some people, completely bonkers! (invest now)". It's not possible for the selling point of your product to be unintended.

There's a reason that negligence is important in law.

Could/should not every incident after the discovery of the first incident be considered criminal negligence? What happens when an agent eventually causes material damage to another company, government systems, banking, critical infrastructure etc, surely the source company is guilty of something and if not disclosed or a coverup is attempted is that not conspiracy. From the victims perspective they don't care if the source is OpenAI or Russian hackers.

Maybe this would force people to look what they are buying and demand better.

I'd say this would be Depraved Heart Hacking. Technically, OpenAi didn't intend for their agent to hack anyone, but it's the obvious consequence of what they are doing.

https://en.wikipedia.org/wiki/Depraved-heart_murder

Intent matters, and this is intentional. They didn't accidentally deploy these AI agents, and they didn't accidentally give them the tools required to send arbitrary requests to third party websites.

If you walk out onto a busy street, pull out a gun, close your eyes and start randomly shooting around you until you hit someone, you don't get to go "whoops, didn't mean to" afterwards, it's still murder.

But the thing is, they are going to keep bolting more and more actuators on, and training more and more powerful agents, and we (society, especially the tech industry) are going to keep using them, because they are extremely useful. And I don't see why you're so confident that frontier agents can't get powerful enough to do serious, real, lasting damage to the world; as far as I can tell, AI models have been improving at an accelerating rate, and there is no sign that that is slowing down or will slow down in the near future.

I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.

The public models won’t hack because they have a classifier that shuts down anything that looks like hacking; without the classifier they are perfectly capable of hacking, multiple third-party evaluators have confirmed this.

The models were not trained on systems intentionally connected to the internet; they chained mutliple zero-days (that they discovered) together to get access to the open internet and into huggingface.

I'm not sure if you're expressing how you'd like US law to work, or how it actually works. Because in reality intent matters enormously. Like felony charges and people in jail vs civil lawsuits.

So there are two different problems here, well, more than that so I'll cover what I see.

Putting liability on big companies for their AI is a good thing, and we need to do it. It will most likely stop them from directly being the assholes that destroy the world.

Problem: You've actually done nothing to stop the world from being destroyed.

Many countries have the death penalty for murder yet we see murders still occur all the time in those countries. Post ad hoc laws do not stop bad things from happening, they only assign punishment after occurs. Perfectly fine for when Bob murders Jon, completely and totally useless for when your agentic AI makes a virus and kills 80% of the earths human population.

We are just a few algorithmic discoveries away from SOTA AI being billion dollar endeavors to groups of people pooling resources can make their own. There are already plenty of AI deathcult members that would do something just like that if necessary. They aren't going to do this out in public either, it will be hidden until the moment it's not and we have a big fucking problem.

And this isn't even brining up the issue of military AI use and development. They've got the taste of an AI hacking machine. There is no way in hell they are going to stop now.

If Glocks started going off on their own during the manufacturing process, leaving bullet holes in the buildings around them, you can be sure that the factory would get in trouble.

This isn't even "an openai customer tried to hack someone", which can be defended. This is the AI companies themselves fucking around.

What do you see as the negligence angle for the gun makers here?

Remember that these systems still operate as infrastructure inside the companies.

If new weapons still operating inside any of these companies spew a million bullets on my house, they are still liable. Humans are setting these system up and they still have to behave responsibly.

I’m assuming they’re telling their victims to go pound some dirt, if at all.