Hacker News

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people).

Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.

> I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?"

You let a second LLM supervise the first, and don’t give the user/customer any way to send information to that LLM.

For example, you can run a LLM trained to do sentiment analysis on the responses your customer chatbot generates and filter out responses that are impolite.

You also can run one trained to flag potential legal issues, thus ‘preventing’ your chatbot from making the wrong promises to users.

Is there any good tech for it, though? This just seems like an inherent language model behavior and at best everyone has guard rails or big exclamation marks to separate their own instructions a little.

It’s a language model. The spoken and written language we use mixes code and data and requires judgement, experience and intelligence.

It’s insanity. We’re fucked.

This is the methodology I use.

No determinism, no separation of data and instructions, centrally controlled.

What couldn’t go wrong?

That's precisely why I am using a different analogy when talking about this. The SQL injection analogy only matches the injection part, not the rest. There is nothing to secure, because there is no SQL query. You want the agent to work on data, in a "general" way, otherwise you'd just use a script.

The better analogy is phishing. Because that's what's happening here. The "prompt injection" attack is trying to "phish" the LLM into doing something unintended. That's how we should all comunicate it, as it matches better with what's happening. Unfortunately there aren't really good defences for it, as we all know from phishing "education" / "campaigns". Your best bet is to secure it in layers, try to have warnings (i.e. classification models) you try to secure the next step (i.e. capabilities based tool execution) and so on. But it's not foolproof and it should be communicated clearly.

> almost fix SQL injections everywhere

Oh if I had a euro everytime someone claimed that.

No, you're still just one clever prompt away from getting pwned. It's like trying to solve SQL injection by attempting to use an ever-increasing pile of regexes for "input validation", rather than just getting rid of string concatenation and using prepared statements instead.

If you read the article, you can find out!

It's bunq. It was time to close your bank account with them a long time ago. Terrible working environment, terrible leadership.

Count yourself lucky if they don't hold your money hostage.

Would it be simple to explain as well? I'm interested

Thanks for chiming in.

I agree this is not a one-click account takeover.

But I think point 2 is broader than that. The user does not need to ask about the malicious transaction specifically. Any normal question that makes the agent fetch recent transactions could bring the attacker-controlled text into the LLM context.

Unless I missed it they didn't provide any proof of this actually working. Really seems like a thing veiled advert for their product

Depending on how much access the AI agent has, there are worse things to inject it with than a link.

People already click suspicious emails that ask them to login. At a high number of attempts, some chickens will be caught. However, people are now weary of emails since there is a lot of phishing there. On the other hand, the AI assistant env. could be considered "safe" by users because it's stuff coming from the bank. So they are more likely to fall for it. (honestly, unless you are a dev and aware of prompt injection, I don't see why the users wouldn't fall for it).

I think the critical part is that it launders an arbitrary URL as trustworthy. The alternative is “Don’t trust anything our bot says at face value, please.”

I think a better criticism is allowing arbitrary text (including URLs) in a transaction description.

Something my ex never understood