Hacker News

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.

I myself had preferred to use MySQL because it was so simple and easy to get started and using it.

Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.

Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.

I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all.

What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases.

For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.

CYA. This approach not working is "the user's fault."

It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?"

It's the easy and cheap way out.

What would a serious security model for an agent even look like?

I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.

But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.

I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.

There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".

Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.

In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop

The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.

That is exactly the reason why we have lonters, errors and limitations when programming.

Bc the "give a check by hand" or the "unbound discipline" never works.

The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.

Yep, file it in the same folder as "Terms and Conditions" notices.

As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up.

In fact it might actually be the solution that works.

Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.

Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake.

Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).

The pattern resurfaces because the prompt sits where it is cheap to implement, not where the question is answerable. "Allow rm -rf /tmp/build?" asks you to predict a consequence from a string. The thing you could actually judge - did this run produce the artifact it was supposed to - only exists after you have already said yes.

I ran an autonomous pipeline in production for eight months and logged every silent failure. Two that make the point:

- My top-level health signal stayed green for three days while zero artifacts shipped. Sixteen daemons alive, backend responding, auth token valid. Every signal it polled was true, and nothing measured the thing leaving the building. No approval prompt anywhere in that chain would have fired, because nothing was attempting anything dangerous.

- I wrote 29 quality gates, tested them, committed them. Not one was ever called - nothing was a runner. The unit tests proved the gates worked; nothing proved they were wired. A permission layer has exactly that failure mode available to it, and it is invisible from the outside: a policy that never denies looks identical to a policy that was never loaded.

So the 1-in-3 miss rate reads to me less like inattention than like being asked the wrong question at the only moment the system offers to ask one. Approval is a claim about the future. Verification is a claim about a result, and only the second one can be checked afterwards.

Genuine question for anyone running agents with approvals on: when did your approval flow last block something you would have regretted - and can you tell that apart from it never having fired at all?

This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.

Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.

The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.

Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.

The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.

The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.

If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!

We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.

If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.

> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.

Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.

That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised. I try to cover both of the previous points in the article and they were some of the most interesting learnings for me

As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)

Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.

The point is more that it's non-trivial and should not be treated as easily accomplished.

In my most recent job I was definitely under that sort of timer.

The most fundamental flaw in the test is that we know we're taking a test.

How many devs take this adversarial a stance to their work?

Also a lot of them may or may not be safe depending on additional context that you don't have in the test.

In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?

Getting behind the wheel of an F1 car on a track involves lots of proving time that you can actually handle such a vehicle. Meanwhile anyone with a credit card can grant an AI system to impersonate their access as a starting point.

I’d give it 2 months for it to turn into a “please drink verification can to continue”-type situation.

Ah yes sandbox it because Docker has never experienced a CVE.

Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.

In a lot of cases there is, but you have to be aggressive about allowlisting commands. It can also be difficult to predict when being able to do a read-only command goes from safe to part of a vulnerability chain.

Also the permissioning system for Codex and Claude Code, while not useless, is insufficiently expressive for a lot of tools which are safe if used a certain way, but unsafe otherwise. For example, the 99% use case of ripgrep (searching for text) is safe, but using the --pre flag makes it able to run arbitrary code. Both of their permissioning systems cannot block flags at arbitrary positions though, so you have to resort to either hooks or aliasing if you want to do this.

I’ve even had plenty of situations where the command was so long that it gets truncated. Maybe my screen wasn’t big enough but as far as I could tell it wasn’t possible to read the whole thing. “Send it, claude!!”

Look at how SELinux is structured, or AppArmor. Neither one is enough. I.e. you need both: file access permissions and permissions to run commands and more... Trying to restrict to only one security feature will make the system either too restrictive or too fragile or useless.

Yet the AI can still escape the "sandbox", unless it is physically unable to connect to another computer and completely airgapped.

Some people at my company take it to the extreme and let Codex run unattended overnight, bypassing permission for all commands. Running on the host, not even in a container or VM.

I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".

1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).

One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).

The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.

Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.