Encrypted reasoning that OpenAI, Anthropic and Google hand back to their API clients can be read by feeding it to a weaker model from the same provider, according to a paper posted August 10. Decoding 315,320 of those blocks scraped from public repositories recovered 367 pieces of personally identifiable information and 182 credentials.
Frontier providers stopped returning chains of thought in plaintext to protect their intellectual property and limit what leaks out of the model. Rather than hold the traces server-side, they return them to the client as encrypted blocks that the client passes back with each subsequent request. Those blocks, the authors write, are “fully compatible and interchangeable across different sessions, users, and models within a provider’s ecosystem.”
That compatibility is the flaw. Injecting an encrypted trace from a capable model into a weaker, less safeguarded model from the same provider makes the weaker model print the trace verbatim. The capable model is never jailbroken and the encryption is never broken.
The eight authors, at institutions including the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems and MATS Research, collected 6,708 agent trajectories published on GitHub and Hugging Face that still carried reasoning blocks, produced by Claude, GPT and Gemini models. Of the 315,320 blocks reconstructed from them, 1,028, or 0.3 percent, held at least one privacy leak. By session the rate was higher: 328 of 6,708, or 4.9 percent. From genuine user sessions the recovered secrets included 62 distinct API keys, 33 passwords, 24 access tokens, seven private keys and 30 personal email addresses.
One trigger recurs. When a user asks an agent to anonymize or clean up a session, the model re-reads the full history inside its hidden reasoning and restates there the values it has been asked to strip. Of 704 artifacts recovered from genuine sessions, 64 appear nowhere in the visible chat at all.
The paper sets out four uses for the technique. It circumvents anti-distillation measures at all three providers. It extracts private data from traces other people have published. It surfaces hazardous material the model worked through internally even where the visible answer refused the request. And it lets an attacker hide a prompt injection inside an encrypted block, where it is invisible to anyone who later reuses the trace.
The underlying interchangeability was disclosed in May by a researcher the paper cites as Green, who reported that providers did not acknowledge “any security implications arising from side channels or replay attacks.” The authors say they gave full technical details to the model providers, Microsoft and Hugging Face before publishing. All acknowledged receipt, and the attacks stopped working afterwards.
Agent tooling has been the subject of a run of disclosures. Researchers split a malicious instruction across tool channels to roughly double how often coding agents obeyed it, and a single email has been used to plant a false memory in an AI assistant. The anti-distillation measures the technique defeats have their own history: the White House warned of industrial-scale campaigns to distill US frontier systems in April, attributing them principally to entities based in China.
The authors propose keeping reasoning traces on the server.
Sources: arXiv, The Hacker News
–
By the Control Plane Editorial Team