KV Cache Residency Breaks Agent Cost Models by 3x

The Broken Assumption

Most teams price agentic workloads like chat: input tokens × per-token rate. For agents that math is wrong, and wrong in a measurable way. The dominant cost in a multi-step agent isn't compute or egress. It's KV cache residency, the GPU memory held between tool calls so you don't re-prefill the context.

Here's what actually happens. Each step re-sends the same long prefix: system prompt, tool schemas, prior observations. If the cache is warm, that prefix is free. If it evicted between steps because the provider's TTL expired, you pay full prefill again. Most hosted providers evict at ~5 minutes. DeepSeek's disk-based cache persists for hours.

On a twelve-step agent with tool outputs, KV cache residency is the line item. On short chats it's rounding error. The agentic cost model most teams use is broken in a specific way.

The Numbers

A DeepSeek billing screenshot one operator posted this week shows $1,050 in actual spend against $3,351 in cache savings. That is a 3.2x gap between per-token sticker price and effective cost on a single account. DeepSeek V4 Pro's hybrid CSA/HCA attention also reports a 10% KV cache size reduction at 1M context and roughly 4x lower inference FLOPs at long context. Cheaper per GB and longer-lived compounds over the hours of a real agent session.

MoE Makes It Worse

The second failure mode is Mixture-of-Experts cost modeling. A 49B-active MoE inside a 1.6T total parameter model has two cost drivers that don't collapse. Active parameters set the matmul bill per token. Total parameters set the minimum memory footprint and shard size. Most TCO spreadsheets fold both into one FLOPs figure, which mispredicts both latency and unit economics. Two models that looked competitive on per-token pricing were not competitive once cache residency was charged honestly.

The Fix

The corrected model has three components:

1. KV cache: price by GB-hour at the accelerator memory rate
2. Active parameters: price by FLOPs per token (not total parameter count)
3. Shard footprint: price by minimum deployable instance forced by total parameters

Then run an actual agent trace through that model, not a synthetic benchmark. The ranking of which models are cheapest for agent work will change. Independent writeups keep landing in the same place: on long-horizon agent traces, cache residency dominates per-token price.

The Harness Multiplier

Token-efficient harness design amplifies these savings. Hugging Face is shipping concrete patterns: agents.md files that front-load context an agent would otherwise scrape from docs, and token-efficient API responses that strip verbose JSON envelopes. Every token saved in a response is a token that doesn't compete for cache space. Smallest stable prefix across agent steps wins, not lowest per-token rate.

The Pattern

The new agent security primitive worth naming this week is the Planner/Executor Split. Two LLM instances. A hard privilege boundary between them. The planner sees the tools and the plan, never untrusted input. The executor sees untrusted content, never the tools. Prompt injection lands on the component that has no authority to act.

This isn't a prompt engineering trick. It's privilege separation ported to LLMs, the same mechanism Unix uses to keep user processes out of root. Gmail already runs stacked defenses on this pattern in production.

Skip the planner/executor split and every tool call is one clever paragraph away from exfiltration.

MCP vs Skills: A Security Boundary Decision

MCP-versus-Skills gets pitched as a bake-off. Read the capability surface and it's a security boundary decision. The asymmetry that matters:

A Skill that ingests untrusted content and shells out is a prompt injection to RCE chain. Not theoretical. The split I'd ship: MCP for anything touching production. Skills for dev-facing agents in CI runners, ephemeral containers, and workstations. Nowhere else.

The Defense Stack

The complete taxonomy has two layers. One makes injection harder. The other caps what happens when it lands.

Model-level (probabilistic — makes injection harder)

• Spotlighting: wrap untrusted text in control tags (<UNTRUSTED>) and tell the system prompt to treat tagged content as data only
• Instruction Hierarchy: fine-tune the model to rank system prompts above user messages above third-party content

System-level (deterministic — caps blast radius)

• Planner/Executor Split: hard privilege boundary, 2x inference cost
• Least-Privilege MCP: minimum operation set per server, typed schemas that reject unexpected params
• Human-in-the-Loop: checkpoint gates on irreversible actions

The Planner/Executor Split is not free. 2x inference, coordination latency, and a handoff protocol you have to design and maintain. For any agent that reads external content and takes write actions, it's the only pattern that stops a single injection from doing damage. Pay the tax.