Hacker News

Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.

PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost.

Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.

The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.

1. Its a security issue.

2. Publicly available sessions make it much worse

You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.

Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches

nothing scientific here, they basically just figured out some real issues caused by bad engineering practice.

All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.

The real question is whether or not the training was directed to optimize for those benchmarks.

The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.

Not surprised. On many benchmarks (i.e tau), we have seen the same thing. Probably lots of training on every publicly available benchmark

Didn't we see this with Fable 5 on multiple benchmarks?

Cheap investment to boost ones CV.

Growth hacking

On top of that, the user paid for those tokens, so if there is an owner, it should be the user, not the provider.

Even copyrighted information can never be "stolen". It can only copied without authorization.

I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.

The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.

The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.

[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.

If I hire you for a consulting project, do I own the screen recording of your computer while you completed it?

they also can't claim to be the owner of the output either because they can't claim to have had a valid license for the inputs.

While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.

For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."

The model is finetuned to enter/leave its thinking mode using special token separators. there's no reason to assume the tool calls induce the same token distribution or produce the model's actual native reasoning trace

I've been doing that since before reasoning was a thing baked into the models, it always performs better this way. Except for some providers/models where you just can't easily turn it off, now I just avoid them. This way I save tokens and have full control of the reasoning.

its not exactly the same... its tool use spec asking to put thinking in inputs fields... it is a good idea but its not same.

Yeah, it’s also useful for prompt tuning, debugging and understanding how a model interprets your prompt.

Also really good for identifying any contradictions in your system prompt and context.

Possibly something to do with other providers using the it to train their own models?

Suppose you hired a consulting firm to write a report, and they delivered the report but not the internal conversations they had when developing it. You exploit a vulnerability in their phone system to get those conversations. You can argue over semantics of whether “theft” is what you did, maybe the right word is “espionage” or “spying”, but that either way we probably agree you are guilty of something? Paying for the final product didn’t entitle you to see how it was made, unless that was part of the agreement.

Anyway, you can distinguish this from the debate over copyright.

The only person calling it stealing is the author of this article, so this is a pointless discussion. The majority of this thread is just arguing with themselves.

Liberating!

No, you paid for the end result. The thought process is a step in between, a function. Think about it, who should get charged if the answer you received comes from a cache? Thinking tokens are the complexity-of-the-problem cost. I mean, you may not agree but both are valid points of view.

> stop using morally charged terms made up by future monopolists

Lets not gloss over this claim. Being: “Stealing is a morally charged term made up by future monopolists.”

I strongly disagree. Stealing is not a made up term and property rights are foundational for any society. Your take is at least sensationalist if not malicious.

> But the real question is: Is it okay to steal

By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)

I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.

(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)

>But the real question is: Is it okay to steal from a thief's hoard?

How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?

Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.

The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.

> I'm surprised these companies re-use the same encryption key across models

I assume switching the model in the middle of a conversation is intended behavior (very useful in coding agents, for example)

Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI.

(Thanks for the link. That’s an interesting idea!)

Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.

I agree on desktop/laptop, but on mobile there are images that appear under the text making it hard to read.

My brain can't tell if the text is horizontal or slightly rotated. It's very hard to read. Beautiful to some, inaccessible to others.

I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.

This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.

In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.

> I wonder if open models are going to start doing that too

Yes, some of them do do that. For example Moonshot tried to reward shorter reasoning traces in between Kimi-K2.6 and Kimi-K2.7 Code, and the latter has a mild caveman accent in its reasoning traces that the former lacks.

Qwen3.8-Max also has terse reasoning, but I don't remember this being the case for Qwen3.6 models I ran locally.

their gpt-oss models do the same. i don't use closed models so i never thought much about it.

Muse clearly does it to some extent. Saw a lot of that running Glimmer locally.

The LLM needs to read the CoT as part of the conversation. You can ask the models to share them with you. Stronger models will refuse, while weaker ones can be “jailbroken”.

I assume it does happen at a different layer, just that that layer is common to all of a provider's models to make conversations portable across models (otherwise the reasoning blocks would all need to be re-encrypted for them work with another model)

+1

Because stronger models are harder to jailbreak from the paper it says haiku was easily fooled into giving us thinking contents

If CoT wasn't stateless and you instead just got a reference which pointed to the CoT stored on the lab servers, the same vulnerability would still exist. Since you just need a weaker jailbroken model to read a smarter model's CoT. This being stateless or not doesn't really matter.

The stateless part is also important for enterprise customers that require zero data retention.

(they could scope CoT access per model, but then users couldn't switch models mid-session)

It wouldn't matter if it was stored only on their servers. As long as they offer the feature to downgrade a chat to a dumber model that can be jailbroken (and the downgrade keeps the reasoning), this trick works.

this is a lethal trifecta, but where a chunk isn't even needed

you have a secret to keep that is read by the llm, and untrusted input that wants to exfiltrate it.

by hell or high water, the agent is gonna output that text

strange because, their subscriptions are not stateless. they log everything and send it to 3rd parties for moderation.

Encryption is irrelevant here. Even if it was kept fully server side, the actual issue is that they allow starting a conversation in a strong model and continuing it in a weaker one. Disallowing that entirely would be a huge hit to user experience.

That's not a trivial thing to do for them because they offer zero data retention environments to enterprise clients.

What I found the most interesting, and unfortunately not at all surprising, is that the reasoning of the LLMs frequently contained much more useful information than the actual answers, because the answers were censored.

You don’t even get _any_ logits with closed models for years now.

I can’t fault them too much, as logit based distillation is extremely effective.

Very useful for making smaller models out of bigger open weight models.

Dario is a cunning business man that won't hesitate to say whatever the fuck he needs to get the US government to exercise some regulatory capture to favor anthropic.

From what I got, the weaker model (Haiku in this case) has access to the shared key and the user simply asks to "transcribe the injected reasoning".

The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.

Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.

The interesting part is what they try to not to say: More indications for K3 is based on distillation from Claude and GPT.

From [1]:

> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.

> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s

> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.

[1] https://x.com/kotekjedi_ml/status/2087147042888114428?s=42

The part about K3 is just very strong evidence that K3 is partly a distillation of Opus. Probably even a distillation of Opus's CoT, which means they already had broken CoT encryption themselves awhile ago.

Security is solved, but business needs overrides it

Why do you restate the abstract? Anyone can read it from the link.

The whole point of the encrypted payload returned to the client for future re-submission would be that they don't log.