Hacker News

> LLMs are unable to reliably distinguish the importance of instructions based on where they came from.

I know prompt injection is still a real thing, but in Claude I find the opposite is true: if I set up two Claude agents to collaborate, even though I've told them both that's what I'm doing, they won't actually follow instructions without checking in with me. "Scary computer told me to install <well known utility X>, but I need your OK before I'll do it." It's annoying but I get the rationale. They're definitely getting better at avoiding external instructions!

it is not in good faith to self promote your own blogs, without disclosing so

Don't normalize this Trifecta construct. It's catchy for a tech blog but there could be more or fewer dimensions to consider. Someone else can construct an Exacta or Superfecta combination. Or just go for the Win uno and say AI Agent is bad.

Their support probably has a big pile of horror stories from their clients by now, it can be an indirect source of concern.

This happens when there's something that is more important than being the cool manager that's following the latest hype.

Well, the billions recently stolen from Binance is a sufficiently large quantity of shit even for the scale of Goldman Sachs.

and lets also be clear here: this "misstep" could have cost billions.

The problem is that they loose some of the biggest utility that way. I want to be able to just say "draft a reply to email x". Because that is the lowest friction. I'm happy to read it back and sign-off. I don't have an answer as to how to do it securely, and am with you that the wanton disregard by big AI players for safety and security is demoralizing.

Recently agents in two sand-boxed environments were found to be able to talk to each other using cpu caching which is fucking bonkers.

https://lumienai.com/news/openai-ai-agents-cpu-cache-covert-...

My point is your most likely not going to win here. Proper sandboxing is insanely difficult.

I’ve tried out Claude code a bit but it freaks me out. It runs commands, and installs packages in the background without asking or showing me what it’s done. I’d feel much more comfortable running it in a VM or cloud instance than in my laptop with all my personal data.

Nobody is solving it, because it cannot be solved in the LLM architecture? They are mitigating it.

We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.

Yeah, I'm working on an open source product in this space [0]--particularly in finance--that's why I know about this problem quite a bit.

I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.

It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.

[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed

We do it through a central ai tool server, openAI compatible interface + MCP, where any knowledge ingested is still linked to the source (always good) and the authorisation layer of the source that was ingested. If we consume a wiki where permissions are managed with a Windows AD, this info needs to be collected when you ingest the information. Such connectors are source specific and must be aware how the system manages permissions. With the correct architecture, you

That way any agent tool only has the information the employee was able to access anyway. A larger problem is to ensure quality control for any answers with different context depths, but that isn't easily testable with or without access management.

It isn't product-y, but often the standard RAG-AI pipeline that is also used to delegate queries to either local or remote agents (and the latter might also not be allowed to consumer PII for example - access management on the other side)

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

Stuff like Kong with their API/AI/MCP Gateways (GS use Kong at some level internally as they have done talks on Kong).

And then they will almost certainly have some form of SWG/SSE tool and endpoint/network DLP tool to block stuff.

I’m building an AI governance framework within a self-hosted AI agent platform designed for these scenarios: https://github.com/rush86999/atom.

I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.

I've been building an MCP server and an AI Gateway as two separate projects designed specifically to prevent agents from doing effectively anything programmatically, rather than trying to convince them with a clever prompt not to do a bad thing.

My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.

One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.

Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.

https://advancedaisociety.org/proof-of-control

Basically every IT device management firm out there is working on an AI governance product.

I think bankers don't have staging/production/codespace. They have an excel with real data, they give it to Claude and say 'do your thing'

Cloud agents or even sandboxes are too technical for most. And most corporate workstations do not allow sandboxing of any kind (virtualization or containers).

It’s the case in all industries. People have been feeding medical patient data into Claude

Finance is a whole industry and you will find all sorts of approaches, certainly including letting Claude run amok with your credentials.

You can certainly imagine very senior and powerful traders at GS demanding it. I'd be only slightly surprised to find that GS allowed it. The traders would have had more power than the IT people trying to forbid it.

Chad is surely using tmux...

Remote-hosted agents are available in most places. BUT they are not interesting, because they do not have access to all the user's data.

People really want to run agents on their data.

Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.

The article is referring to the dilemma of the trifecta twice.

Seems hard to do well in practice.

Can the agent read the internet for context? What if the fetch reveals a secret?

> Goldman has control of those desktops and could ensure Claude Code is always sandboxed

Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?

I got a series of Teams messages from some random person at the work address yesterday asking me to fulfill some access request to them. I noted by the part of their org (wildly different from mine) that they were looking for a different person.

I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”

Read into this what you will.

> they allowed that exact situation for a while now and no one cared

Source for them having allowed it versus not tightly enforced IT rules?

This is global, starting with Hong Kong months ago. I doubt fear of US regulators were was the motive.

ELINA, please! As a non-American, I don’t understand why anyone would care about Trump in this context. Aren’t the people who look into these things different departments? And don’t they only look into anything after a breach?

Subscription plans are limited to companies with fewer than 150 users.

Anything above that requires an Enterprise plan, which is billed at API rates.

For businesses anthropic offer a teams plan for up to 150 users which is billed like a collection of individual users i.e. you have a seat, you get some tokens/month, you buy a higher tier seat and you get some more tokens/month. For >150 users you pay $20/month/user just for access to the platform, then all tokens are billed at API rates.

Those personal plans are hard to manage. As an administrator I can't control settings across the org if everyone is on their own plan which means people may be giving sensitive code/data away to Anthropic or sketchy connectors without my knowledge.

So most businesses at least use the Anthropic Team plan, which is $25 or $125 but isn't as generous as the personal max plans. Overage is API rates. Almost all of our devs are over the allotment most months.

Side-anecdote... I have a friend in R&D at one of the big chip companies that claims he racked up a one month 20k Claude bill and has been told to step it down a notch.

Large companies can’t use those plans. The consumer or single dev plans appear highly subsidized based on the difference between rates for the same thing.

Hah I can’t imagine only using a $20/mo plan. (Robotics engineer)

I thought enterprise contracts (ie teams with >x users) offered only API prices. Am I wrong?

The enterprise cost controls that Anthropic offer are blunt, really only allowing for allow/disable specific models and a general monthly hard cap per user. No per-model spend allowance. Their "reduced cost" mode is just Sonnet on Low with no flexibility.

They are frankly reaping what they sow here, enterprises have been very clear about what controls would help us align costs, but they have been very poor at shipping features on that front.

You aren’t allowed to use the flat rates for companies, unless you’re cheating them they’ll come for you eventuallt

> The $20/mo plan ...

Is just the first hit your dealer gives away for free. Treating that as the actual baseline cost for what they need in order to justify current AI valuations requires some truly fraudulent mathematics. Hell, even the current API rates are unsustainable -as-long-as- they need to keep investing into training the next model they will be serving 6 months from now.

The plan has always been to develop these models using borrowed money, get everyone's business process addicted to them, moat off access to alternatives (e.g. chinese / foreign open-weight models), via technical superiority -or- more likely regulatory shenanigans, THEN jack the prices up through the roof starting with direct enterprise customers buying API access but then obviously bubbling down into literally every product / service every human being on this planet purchases. Unfortunately, it's a garbage plan based on a multi-ton house of cards that is about to collapse on everyone.

Yes, in industries where you have legal/regulatory/security restrictions on your data, you pretty much have to pay API rates. Furthermore, if you buy through providers like AWS Bedrock or GCP Whatever-The-Fuck-They-Renamed-Vertex-To, depending on your setup you may not benefit from the pricing on cached tokens. It can anywhere from 10x to 100x the prices you pay.

Most folks I know have companies that deploy the models via Bedrock or Azure foundry and the token cost goes against their spend with those vendors.

Anthropic forces you onto api billing for orgs above a certain size

Yes, any decent size business is paying API rates.

Enterprise plans are billed at API rates, with small discounts only if you’re quite big.

Yes, many companies have Enterprise, for SSO/SCIM, audit logs, data retention, HIPAA, support, org-wide connectors and skills, usage tracking.

That is usage based and therefore expensive.

Putting everyone on their own plan is economical, if you can give those things up.

At the enterprise level, API billing is post-paid, and a lagging indicator on an unlimited expensive resource with bureaucracy between service owner and end user is a recipe for disaster

Honestly, for most casual business work (document analysis, data science, web scraping, basic scripting, etc) the Gemini Pro models are pretty well rounded and competent.

The problem is we now have a two tier infrastructure where some teams get Claude and some get Gemini and IT has so support both.

Anthropic does not sell subscriptions to organizations over 150 people, only API billing.

Companies this size don't get the subsidy

their subscriptions aren’t subsidised. in fact its near 100% margin. you are getting the lowest priority access with no service level guarantee in the gaps in supply left over by api users, anthropic’s cost basis on hardware would be the same without subscriptions.

Extending the thought, there’s also the risk it enters critical path but something upstream changes or Anthropic starts charging 1000x for the tokens and now someone needs to walk through god knows how many lines of never before reviewed code to figure out how it works.

By wholly depending on the assisted coding you are wholly dependent on Anthropic, forever, for something that is now critical.

There are probably thousands of these little microscopic dependencies that occur when you make these tools widely available. What happens when you don’t want to foot the bill anymore?

This is what happens. We had this happen with end user scripting pre AI where these scripts became unknown production applications. It made the actual engineering impossible because you had no idea if anything you changed would impact the ??? processes run by users that have become critical. AI would amplify that greatly so we have an explicit rule that it can only be used for reporting and prototyping only and we restrict any write action

Isn’t that a good thing? It would be irresponsible to introduce non-technical students to building software without at least covering some basics like a terminal and encouraging some understanding of the limits of their capabilities.

Elephant in the room is QA, the thing that disappeared when Agile took over the world.

> Pre LLMs, the main barrier to producing software for non-tech people was writing the code, literally.

Based of my several rounds of discussions about every features, non-tech people usually have several unknown unknows about their idea. Their requests are usually one sentence or one paragraph, and then to flesh it out requires several back and forth which usually start with “I didn’t think about that”.

It’s like how everyone has seen people draw, play music, or build an house. But when the time to do so, they’re hopeless to replicate it. Because what you see is just surface level.

> But GS is saying they shouldn't leave their sandbox.

Exactly

Technically yes, but in practice:

a) They are pulling it entirely rather than imposing a cap or enforcing the use of cheaper models. This would be a strange choice if cost was the only concern.

b) Famously, traders at Goldman are well renumerated. Given the costs of these people even small productivity improvements carry a large value.

c) It would be very difficult to determine in any kind of conclusive way that the benefits were smaller than the costs.

For all these reasons I find it totally implausible that this decision was driven by API costs.

Definitely not using something like an Anthropic model for research, FWIW. The amount of data the LLM needs to consume to build context for even simple trade sequence comprehension makes it next to useless.

There are a litany of other mathematical and ML models that can more aptly be applied to the space.

But to answer your question more directly, yeah...there's no way in hell you'd let the questions your traders are asking in research out of the four walls of your building if you can avoid it.

I have bad news: most large corporations are successful because they know how to make money. Often, the internal operations, are disappointingly bad, just not bad enough to kill the corp fast.

Source: I've worked in and with many successful organizations, and worked at one place with proper internal processes, so I know what's possible

I'm no longer in that industry, but as I recall there are actually rules imposing independence from investment banking and research arms, for example. Dates back to the Dot Com era where research analysts were touting IPOs for the investment banking division.

In this case, both decisions are weighty enough that they should be made at the very top, by the same people.

> In an honest, well-functioning org, those are decisions made by completely different groups that do not (and should not) report to each other.

I don't agree at all. Different groups should be supporting each other and the overall bottom line instead of their parochial, narrow self-interests - the latter is a poorly functioning org.

If by group A delaying its $100M cost-cutting, group B can add $1B to the bottom line, it would be a corporate and group management failure for group A not to cooperate.

Because it would be surprising for GS to make inappropriate choices?

Google put 'scandal' in the autocomplete choices after their name, that tell me enough.

> A drug dealer never consumes

That is:

> Number four, I know you heard this before

> Never get high on your own supply

Nonetheless, I think the advice being applied here is number 3

> Number three, never trust nobody

They were sacked for vibe-working, which is how AI is supposed to be used if you listen to some business hype. Of course it is not how it's supposed to be used according to people who actually know the current situation.

not this kind of non-deterministic tool though.

Not to forget versioned inside the document. The same spreadsheet produces same answers if it is not edited. Same can't be said from programs that can be recreated over and over again.

It's funny you mentioned excel macros not being malware. Various scripting systems provided by excel has been a long standing malware vector.

> don't cost an extra engineer's salary per engineer.

In fact they preserve an engineer's salary. Many front-office wranglers in finance have built rock-solid job security using impenetrable nest of excel macros that only they understand. Or was so, until recently. Now we have AI.

Well the real question is: Is this about using AI, or switching from public AI to private AI, or about Anthropic having access to confidential information of their own underwriter?

Look no further than the recent email leak story. The tool is quite powerful and I can see the argument that at an enterprise / risk management level it should only be used by those capable of reviewing it.

Goldman any day, if you're asking based on their respective reputations.

They've been in business longer than the USA's existence, they must be doing something right - some 336 years.

The UK is very good at banking.

Maybe not Goldman.

But DEFINITELY NOT Barclays.

Better not ask that question to Merrill Lynch circa 2006

> Imagine a nefarious banker creates an app that lets him talk to a trader on the other side of the "Chinese wall" about M&A deal flow without getting caught because they are avoiding monitored channels like email, slack, phone, zoom.

Back channel conversations have been happening since forever and hardly need vibe coded apps to pull off. They can use non-corp services on personal devices, or kick it old school and grab drinks at the bar downstairs.

>The ability of bankers or traders to even create an app that others can run, without any IT review/approval process is probable a compliance issue in and of itself.

Now worries! Its called the "citizen developer" today! :-D https://de.wikipedia.org/wiki/Citizen_Developer

Yeah it should be a dialogue with bankers / front office. It can even be "OK if you wanna do this, we gotta teach you at a high level how the computer works, so we can figure out broad company policy together". Done right the rest of company finally gives a shit (because they want to keep their cool new personalized toys), and it's never been a better time to be back office IT.

we are all growing the same thing (internally)

Good security policies always contain building blocks. If your security realies on low-privilage code been fully human-auditted, you've already lost.

Only select high privilege code, where we don't know how to make it "secure by construction", should depend on full audits.

The right use-case here is "custom homegrown Bloomberg Terminal add-ons", not autonomous trading software that can see everything and also isn't routed through trading risk software.

Even managed solutions are just mitigations, the "problem" is coding. Computer programs can do anything. This was gated before by knowledge, only coders could effectively write software, including malware and poor-security software. Now anybody can write software.

The corporate computer is a locked-down environment, the set of actions you can perform are limited. They ideally should be the actions you need to do your job, and nothing more. Any method of writing and executing code breaks that wide open - you can now do anything, given enough time and effort. Now the time and effort floors have fallen, too.

Claude Managed Agents might prevent the agents themselves from doing something nefarious, sure. But that was never the draw of AI. The ideal use of AI is doing the least amount of stuff with AI, and the most amount of stuff with code. Asking AI to sort my email? That's good, but writing a program to do that is better. Asking AI to do statistical analysis of my trades? Again, sure, but it's even better to write a program to do that.

The sensitive data problems aren't just about sandboxing. If a systems development model at Anthropic discovers a clever optimization trick based on a customer's complaints to their account team, that's a great deal that saves a lot of work. If a trading model at Goldman discovers a clever pricing trick based on a customer's complaints to their account team, that's an enormous problem, even if the underlying data is never seen by anyone who Goldman wouldn't trust to handle it responsibly.

They're not pulling Claude from their research teams and software engineers (where the most IP resides), they're pulling it for productivity reasons from employees that they deem aren't benefitting from it (most people), or are just offshoring their work, its not paying itself off for most employees.

Im sure leaking IP is a concern, but they can control that local/not. But yes Jane st has been building internal LLMs/agents for a while.

Old fashioned prediction modeling/ML is a much better use of your GPUs than generalist language models that are basically time-sheet fraud laundering tools.

They're not pulling Claude from their research teams and software engineers (where the most IP resides), they're pulling it for productivity reasons from employees that they deem aren't benefitting from it (most people), or are just offshoring their work, its not paying itself off for most employees. Im sure leaking IP is a concern, but they can control that local/not. But yes Jane st has been building internal LLMs/agents for a while.

Old fashioned prediction modeling/ML is a much better use of your GPUs than generalist language models that are basically time-sheet fraud laundering tools.

I said running a production system.

Sure, they should be able to build some simple tools to help them in every day tasks.

I've been lucky enough to never work with engineers who take offense at rough sketches, specs, or prototypes.

If a team wants the benefits of velocity with AI, the consequence is volume.

To deal with the volume, the system needs more checks than humans can handle. Additionally, the humans no longer understand the details at the low level of implementation at scale. Therefore, the system only needs to assert that it is following agreed patterns.

So the initial scaffolding needs to be agent friendly. The processes need to be aligned to agents. The tooling needs to be agent operable and put on rails. The patterns need to be documented to be enforced.

Otherwise, teams get volume that cannot be handled by humans. So full speed agentic engineering requires teams to be all-in.

A piece of this is to leverage the harness to normalize or standardize output. Standard patterns, standard architecture, standard UI libraries. These are all much easier to enforce with agents than with humans, but requires extensive platform and harness engineering.

Teams do not really understand this yet. Engineering teams still think their job is to build product faster. By the 10th time you tell an agent "implement issue #15639", you realize anyone could have done that. Engineering's job now is to own the enterprise harness (context layer, ephemeral agent env, ingress channels, auth proxies, MCP servers, quality gates, agent ops, etc) so anyone can build the product. That's what "all-in" means.

Once this system is set up and well executed, it really doesn't matter who writes the initial prompt. An enterprise harness designed to enforce safety, security, and best practices applies to a senior dev or a junior dev or a product owner all the same. All-in on engineering the enterprise harness, not the product.

That's how I see it.

You have to go all in on agentic engineering. And if you're all in on agentic engineering and doing it well, then anyone should be able to build safely and no one needs to look at code. The code should look highly consistent and agents should be enforcing best practices, approved architecture, running security scanning, etc.

The gains at the middleground compromise are not really significant and, IMO, creates frustration and friction.

Works best to go all in. GS should let everyone have CC and GH and focus on making their enterprise harness safer for everyone.

>allowed Traders to put in limited C++

This is gold! :-))

If it is just a stateless UI, then probably fine. I've converted a number of these into production apps and the gaps existed in places a non-technical person wouldn't look: the overall architecture, database schema, security, and handling of edge cases to name a few.

If you cannot make your chainsaw safe (enough) to use for the market-share you want, it's not a good chainsaw

You really think that a company that settled for 1.5B for stealing IP is going to abide by the terms?

You also think your emails are gone if you hit "delete" in a web interface?

MD is often “Managing Director” when the top level subject refers to Goldman Sachs, et. al.

Also AIUI both claude[0] and chatgpt[1] offer HIPAA compliant BAA’s.

0: https://support.claude.com/en/articles/13296973-hipaa-ready-...

1: https://openai.com/solutions/industries/healthcare/

They have ZDR arrangements for this reason

Do you think my wife is dumb and doesn't take her job seriously? She is very aware of HIPAA rules and makes sure to steer clear.

Oh she is definitely doing that

There are already HIPAA-compliant AI systems in place. My doctor uses an AI transcription service for her patient interviews.

> Don't be naive. Don't be alarmist.

To me it’s the phrasing “I got her using Claude code”. Maybe that means “I helped her set up access through her employer approved AI account” but it’s not exactly that common for such access to be set up by your husband. It just as plausibly means “she’s using the same personal account she’s using for the family tasks I also mentioned”. I don’t see that concern as naive so much as in tune with the way people frequently use tech.

Thank you, I am taken aback by the crazy reaction here. People assume doctors are some sort of tech illiterate luddites who would be throwing around PII left and right if they weren't restrained by us smart programmers?

She takes HIPAA very seriously, and yes she got Claude Code access through her workplace. I did not set up my personal CC account on her work computer and there is no way she would let me (or her corporate controls even if she did...)

From context I assume: Claude. "It" might fit better?

Claude most likely...