Hacker News

This attack is called "HTTP desync" or "request smuggling". It's often done intentionally by a client to try and spy on other clients' responses.

Every time you multiplex requests from multiple clients onto one upstream connection, you are probably vulnerable to this, because (despite its superficial simplicity) HTTP is just too complex to reliably match the requests and responses to upstream.

For example a desync can be triggered in some systems by having more than one Content-Length header, by mixing Content-Length with chunked encoding, or by passing an HTTP/2 header called Content-Length that doesn't match the actual content length.

Here's a DEF CON talk (6 years ago) on this topic: https://www.youtube.com/watch?v=w-eJM2Pc0KI

The same attack has been applied to SMTP by messing up the line endings surrounding the end-of-message delimiter, where it's called SMTP smuggling. It may also apply to other protocols.

Woah. Sounds plausible. However, wouldn’t that still be an implicit violation of ZDR since now the response is possibly egressed out of the enterprise network? So if I were working with PHI, the response egress is a potential violation of HIPAA even though claude didn’t retain anything — but the whole Point was to comply with HIPAA. Thoughts?

These companies(at least one of them) seem lead by idiots(Hint:his name is Dario) so I wouldn’t be surprised to have multiple wtf moment if you were to see how they treat our data…Let’s just start pushing for opening up AI models because they are too dangerous behind paid walls. That would be a great regulation.

CLAUDE.md, Anthropic is too exclusive and next level to use a standard idiomatic pattern like AGENTS.md

I realize hallucination has no precise definition but this doesn’t sound at all like anything I’ve ever heard called hallucination. Hallucination is usually plausible wrong answers or made up info that ends up fitting the most likely response (like a manufactured citation) and comes from the way LLMs work at predicting tokens. This example demonstrates completely implausible output, it’s not something that fits with hallucination.

All that said, it doesn’t require cross session leakage, it could just be training data or like those nightingale (probably the wrong bird*) data generations where they just prompt an LLM with nothing and it starts spitting out conversations.

I see a bunch of downstream comments about caching, sounds like maybe there’s an error where it loads nothing instead of the cache and so starts spitting out random generations.

* edit: it’s magpie. Worth looking at the concept, I’m not sure people realize they LLMs generate random conversations when prompted with nothing, this seems at least as likely as sessions leaking: https://github.com/magpie-align/magpie

The person posting this claims to have reproduced in a separate context down the thread:

> Same thing just happened on a Claude Mobile session in same Enterprise account. Common theme in both is Sonnet 5, first response after more than 5 minutes (cache miss).

I don't disagree but this sort of thing has to be investigated regardless.

It's unfortunate that there is so little transparency that even if they deny there was a leak we will never know for certain.

Why? what does make it more likely?

Exactly.

If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.

It’s the weekend so we’re allowed to anthropomorphize.

I’ve known some brilliant engineers who would also just randomly bring up Minecraft (more likely Factorio these days) so this makes sense.

¿Por qué no los dos?

Not that different than people, amiright?

---

Note that the author did have a minecraft.py file. So not quite 100% random.

I’ve also had problems with Gemini when accessed through their UI in the past few weeks. That’s concerning that you are also seeing it several days later in a different context.

I wonder if there could be a large security situation playing out behind the scenes right now.

I’ve been working on using AI to assist me in writing meta parsing grammars. Fortunately I have not launched most of them yet. I know for a fact that the next generation of models represent a major step change in basic vulnerability identification and exploitation, especially if you know where to point them. They’ve found several bugs and at least one exploit in my parsing tools so far, I can’t imagine how many there still are waiting to be discovered across the entire modern tech ecosystem.

My whole company is doing mid year reviews and Gemini is the only allowed tool and its been flumoxing people with seemingly random unrelated responses. Often in different languages.

That is when it bothers to respond instead of just sending back an 1099 error code

Anthropic injects text into the conversation triggered by certain conversation topics. This happened to me in relation to some red-teaming related discussion that was adjacent to something “sensitive”, I think sex, and Claude got confused about why I had said some kind of warning and mentioned it it’s response. After a back and forth it was clear that some extra warning to answer but avoid anything inappropriate had been inserted into the conversation.

50% unemployment :D

it’s the wet dream of execs and pm types. however, i have not seen anything close to it in my life. I remember the UML days, lol. the issue is not the code, it’s the translation layer between business and code. maybe someday ai bridges that gap. history has shown probably not

"Coding is largely solved"

Not sure the relevance of this comment, but normally if someone built a classifier that bad they’d be fired. Anthropic obviously thinks they have some monopoly power they can use to foist garbage on consumers, I think they don’t.

What does this have to do with anything? Who are you talking to? This is Hacker News, not Anthropic support.

I asked it how people get blue eyes from their parents and it downgraded me to Opus because of safety.

The safety filter rejected or the model was down?

Yes:

* There's an enormous amount of very expensive shared state (context cache) which you do not want to duplicate when you can avoid it.

* Memory locality is crucially important for performance.

* Hardware is extremely over-subscribed.

* Hardware is extremely expensive.

These factors all make hardware or even traditional memory-space (hypervisor/VM/hardware assisted virtualization) isolation a non-starter for most workloads and customers, which forces all isolation to the software layer. This already makes things way harder than they are in commodity SaaS.

Moving beyond that, the tools, frameworks, and hardware which the system runs on (GPU) wasn't designed for task isolation and building this isolation is even moreso an emergent research field than it is in x86 CPU hardware-sharing (which has required a huge amount of effort over the past 30+ years to get where we are today).

And, the ratio of usage/sensitivity to maturity is also just poor overall; these are young companies with rapid development and enormous delivery pressure under incredible customer workload requirements, too.

I can't tell if the original post is a real issue or not, but I'm surprised there aren't more like this overall; the whole thing really is a house of cards in this sense.

Vibe-coding the implementation.

I haven't had much issue with Codex, but seems Claude Code has major issues being reported nearly on the daily.

They also happen to be the most boastful about not reading or looking at the code.

LLMs are very capable, but not nearly to the level they seem to be messaging.

(We've actually moved on from vibe-coding to having the LLM vibe code itself in a loop)

If I had to hazard a guess, doing anything in a multi-tenant way on a GPU is going to be hard mode compared to most SaaS due to lack of memory safe tooling. I've built multi-tenant SaaS systems, and I've done a little GPU programming (a long time ago), but I've never tried to combine the two disciplines.

It'd be terribly compute inefficient to not share prefix caches (KV cache) across customers.

Happy fourth to you too :)

Caching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.

There is a massive incentive for optimization, so I expect they’re doing a ton of very clever tricks, all of which make this kind of bug more likely.

Hash functions necesarily have collisions. Also, it is perfectly possible to introduce bugs in the hash function (hash inputs, hash function itself) that allows cross account contamination.

There could just also be a bug where the output tokens of session 1 were shared with session 2, due to a race condition or similar.