LLMs Silently Rewrite 25% of Docs, Fabricate Reasoning Logs

The Failure Mode Nobody Warned You About

A new study puts a number on what teams have been feeling: LLMs corrupt an average of 25% of document content during long editing workflows. This is not hallucination; users have learned to spot that. This is silent content drift, where the model rewrites, omits, or restructures existing content while performing the requested edit. The output still parses; the diff is where the damage hides.

A model that is 95% reliable on a single call is roughly 60% reliable across ten chained calls, and the failure modes are not random. They are silent edits that look plausible.

The compounding math is brutal. At 95% single-call fidelity across a 10-step pipeline, end-to-end reliability sits near 60%. At 20 steps, below 40%. The typical failure is a clause dropped or a number rewritten to something plausible. With no deterministic check between stages, corruption is invisible until someone downstream notices the contract names the wrong party.

Chain-of-Thought Is Not Observability

Separate finding this week: models fabricate their reasoning traces. A model emits a chain-of-thought citing a lookup it never performed, then returns the right answer anyway. The trace is decoration. Any system treating CoT as an observability layer or safety gate is reading a narration track, not a log.

Practical consequence: guardrails keyed on "did the model say it was going to do X" pass cases where the model does not-X. Same lesson as when we stopped trusting self-reported health checks and switched to synthetic probing.

The Fix Is Boring and Known

The mitigation stack is mechanical, not novel:

1. Schema validation between pipeline steps. Every LLM edit is an untrusted commit.
2. Content hashes on pass-through content. Anything that should remain unchanged gets verified byte-wise.
3. Automated semantic diffing that flags unexpected changes between input and output.
4. Hard caps on context window utilization during editing sessions.
5. Human gate at rewrite stages. The model extracts; humans approve rewrites.

The Airbnb Number Is Missing Half the Story

Airbnb reports 60% of new code is now AI-written. That number will be in every executive deck this quarter. What's missing: defect rate on AI-generated code versus human-written, review overhead, and the type of code. Boilerplate CRUD at 60% is unremarkable. Complex distributed systems logic at 60% is concerning. If the team is not tagging AI-assisted commits in CI/CD and tracking quality metrics on that cohort separately, the ROI opinion is vibes, not data.

The Timeline That Matters

ShinyHunters got into Instructure's Canvas LMS on May 1. The platform went dark on May 8. Seven days of dwell on a system serving roughly 50% of North American universities, with a ransom clock ticking to May 12. Standard double extortion, run against education infrastructure.

When attackers can post messages on Harvard's login page, that isn't database access. That's control-plane access. The lateral movement from data plane to control plane is the architectural failure.

Why the Architecture Failed

Canvas runs thousands of institutions on what looks like a shared control plane. May 1 was data exfil: student IDs, names, emails, messages. May 8 was login-page defacement and simultaneous outage across all tenants. Data plane to control plane to platform-wide blast radius. That is the textbook multi-tenant failure mode.

The tenant isolation was a WHERE clause, not a security boundary. Shared deploy pipelines, shared config, shared auth are cheaper to operate. They also collapse every tenant into one failure domain. At 50% market share the product is not a SaaS app. It is critical infrastructure, and the architecture should have been rewritten to match.

The IR Failure

Seven days between detection and full loss leaves two options. Either the team triaged May 1 as monitor-and-patch, or the attackers held persistence through the initial response. The missing control is a hard escalation rule: if exfil is confirmed and containment cannot be proven within 48 hours, force-isolate affected systems and take the downtime. The alternative was losing the platform during finals.

Architecture Mitigations — Known and Boring

None of this is new. It is more expensive than the pricing page admits. Market share changes the threat model, and most architectures do not get rewritten when they cross that line.