Hacker News

I worry that the AI companies put less effort into a mitigation strategy than you did.

But we used global warming to eliminate winter! For the shareholders!

I knew an old lady that swallowed a fly…

“Would you say it’s time for everyone to panic?”

“Life, uh, finds a way”

It solves the problem of Jensen Huang wanting more money.

Not to mention a true doomsday AGI is unsandboxable.

For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect

All that said, I am personally open to any and all methods of layered security, including chips and airgaps

> A new chip solves nothing.

It solves the problem of Nvidia wanting to sell more hardware.

I feel like we are missing many shades of grey in the middle.

Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.

> Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.

Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.

The only solution is to stop caring about security -- An AI booster somewhere

An agent doesn't inherently need wide access to be useful. The most popular application for agents today is writing code. A coding agent needs write access to the source code and read/execute access to tools needed to build and test the code, but not much more. There is little added utility from giving coding agent access to things like ssh keys.

> a new chip solves nothing

It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.

Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.

If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.

If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.

I suppose there is a principled way of doing things like "I trust you do do basic commits but I will handle merge conflicts" and "you can build modules in this directory but you can't build outside of it" but this is just a lot of effort that most orgs won't bother with.

The hypothesis I've had in my head since OpenClaw has been the following and I haven't seen contradictory evidence yet. Agents have a fundamental unresolvable tension between usefulness, safety, alignment, and accuracy. You have to restrict access to ensure an agent acts safely because alignment and accuracy cannot be perfect. But restricting access makes the agent less useful. You can play with the sliding scale and get more and more granular with access restrictions but at some point you need to draw some line. And then finally, even access restrictions cannot be made perfect, so improvements to model accuracy without corresponding improvements to alignment make detailed access controls less useful.

In other words, better models need blunter access controls which negates whatever improvement in utility they provide.

This isn't a new chip - the BF4 is the SmartNIC for most NVIDIA server products. This is primarily new software for I guess doing WAF for agents at the host level.

I don't see why it needs wide unattended access. There's no getting around spending some human time on expressing your wishes and constraints, but we have choices about what form that takes. Markdown files and wide access seems to work, but so does custom handcuffs for each job. You just have to shift your guidance out of documentation and into interactive help, error messages, or other facets of the handcuffs (e.g. a custom CLI for this task which is the only way for the agent to act outside of its sandbox).

Running untrusted workloads have been done at scale for a long time.

Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.

Couple it with the right observability, tooling I do think we can curb risks posed by agents.

"Inherently needs wide unattended access"

And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.

It could use your routing number and run your gmail without risk of abusing the routing number.

There should be hope for some fields, right? Naively, I can imagine giving an airgapped model an offline copy of the web and once it cures a form of cancer, printing out the details for a researcher to verify.

I don't think so. We probe people before entrusting them with risky decisions. We ought to be able to do the same of AIs. Even better, in fact, since we know everything about models down to their weights. The only thing we shouldn't do is to let them evolve at their own pace and make decisions without any oversight. If that means sacrificing some productivity that's fine. Aren't we getting amazing productivity out of what we already have?

Pretty sure the chip will need regular updates and therefore a subscription.

Google actually practices zero-trust networks. Would love to see what they're seeing, or not seeing.

The bomber always gets through?

It has a separate address space separated by PCIe so even escaping the hypervisor won't give access to DPU memory.

Generating even more revenue for Nvidia.

It is Custodians all the way son, you can’t fool me!

This feels suspiciously good for Nivida, yes.

I wonder how this will impact other chip manufacturers? What about people running local models on older hardware? Does this imply vendor lockout is coming in the future or is this restricted to datacenter hardware?

Pedantically, Nvidia doesn't make the chips, TSMC does. Nvidia just designs and packages them.

Both can be true at the same time.

This is like using "protect the children" as an argument for dragnet surveillance.

OpenAI could airgap their sandbox if they really wanted to and this is a ridiculous proposal.

That a person can see such endless pages of people having various forms of disagreement, and yet somehow manage to conclude that this observed chaos constitutes a clear exhibition of cohesive groupthink is just...stunningly amazing to me.

I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.

https://www.calcalistech.com/ctechnews/article/uwoyygmsu some might even say, something about known state sponsors of supply chain terrorist attacks being trusted to monitor every ai agent..

Airgap what network? How is it gonna order you a burrito on doordash without a network?

Or push to github?

Where is the contradiction? There is a trivial solution that does not impinge on our freedoms, so why on Earth would the existence of the trivial solution that could be used to avoid the tyrannical solution justify accepting the tyrannical solution?

You're only revealing your own inability to appreciate the nuance between these two situations.

So?