KVCacheResidencyBreaksYourAgentCostModelby3x

◆ DEEP DIVES

1. 01

   Your Agent Cost Model Is Wrong by 3x — The KV Cache Fix

   The Broken Assumption

   Most teams price agentic workloads like chat: input tokens × per-token rate. For agents that math is wrong, and wrong in a measurable way. The dominant cost in a multi-step agent isn't compute or egress. It's KV cache residency, the GPU memory held between tool calls so you don't re-prefill the context.

   Here's what actually happens. Each step re-sends the same long prefix: system prompt, tool schemas, prior observations. If the cache is warm, that prefix is free. If it evicted between steps because the provider's TTL expired, you pay full prefill again. Most hosted providers evict at ~5 minutes. DeepSeek's disk-based cache persists for hours.

   On a twelve-step agent with tool outputs, KV cache residency is the line item. On short chats it's rounding error. The agentic cost model most teams use is broken in a specific way.

   The Numbers

   A DeepSeek billing screenshot one operator posted this week shows $1,050 in actual spend against $3,351 in cache savings. That is a 3.2x gap between per-token sticker price and effective cost on a single account. DeepSeek V4 Pro's hybrid CSA/HCA attention also reports a 10% KV cache size reduction at 1M context and roughly 4x lower inference FLOPs at long context. Cheaper per GB and longer-lived compounds over the hours of a real agent session.

   MoE Makes It Worse

   The second failure mode is Mixture-of-Experts cost modeling. A 49B-active MoE inside a 1.6T total parameter model has two cost drivers that don't collapse. Active parameters set the matmul bill per token. Total parameters set the minimum memory footprint and shard size. Most TCO spreadsheets fold both into one FLOPs figure, which mispredicts both latency and unit economics. Two models that looked competitive on per-token pricing were not competitive once cache residency was charged honestly.

   The Fix

   The corrected model has three components:

   1. KV cache: price by GB-hour at the accelerator memory rate
   2. Active parameters: price by FLOPs per token (not total parameter count)
   3. Shard footprint: price by minimum deployable instance forced by total parameters

   Then run an actual agent trace through that model, not a synthetic benchmark. The ranking of which models are cheapest for agent work will change. Independent writeups keep landing in the same place: on long-horizon agent traces, cache residency dominates per-token price.

   ---

   The Harness Multiplier

   Token-efficient harness design amplifies these savings. Hugging Face is shipping concrete patterns: agents.md files that front-load context an agent would otherwise scrape from docs, and token-efficient API responses that strip verbose JSON envelopes. Every token saved in a response is a token that doesn't compete for cache space. Smallest stable prefix across agent steps wins, not lowest per-token rate.

   Action items

   • Instrument your agent serving path to measure KV cache hit rates and GB-hour residency per session by end of this sprint
   • Benchmark DeepSeek V4 Pro and V4 Flash against your current API provider on your actual agentic workloads, measuring effective per-task cost including cache behavior, within 2 weeks
   • Refactor agent prefixes to be stable across steps — system prompt, tool schemas, and scratchpad should be identical tokens in identical order by next release
   • Build a three-component TCO model (cache GB-hour + active FLOPs/token + shard footprint) and re-rank your model shortlist this quarter

   Sources:On our agent workload, KV cache residency dominated cost · The next caller hitting a given API is not going to be a person with a browser

2. 02

   Three Trillion-Parameter MoEs in One Week — The Frontier API Premium Is Now 6 Points

   The Convergence

   Three open-weight MoE models shipped in one week. All landed within 6–8 points of GPT-5.5 on the Artificial Analysis Intelligence Index:

   Model	Total Params	Active Params	Intelligence Index
   DeepSeek V4 Pro	1.6T	49B	~54
   MiMo V2.5 Pro	1T	42B	~53
   Kimi K2.6	1T	32B	~52
   GPT-5.5	undisclosed	undisclosed	~58

   The remaining gap is concentrated in HLE, TerminalBench Hard, CritPt, and Omniscience. Hard reasoning frontiers. On coding, tool use, and multi-step planning — the actual agent workload — these models are functionally at parity. The 6-point delta that justified frontier API pricing is smaller than the delta between a decent harness and a sloppy one.

   Specialized Models Are Dead Weight

   OpenAI killed standalone Codex this week and folded it into GPT-5.5. The message is not subtle: general-purpose models have reached coding-task sufficiency. Mistral shipped the same signal by collapsing three models into one flagship. Alibaba's Qwen3.6-27B outperforms its 400B+ predecessor on coding benchmarks at 15x smaller and single-GPU deployable. Routing logic that selects a specialized model per task is becoming dead code.

   Stop building routing logic that selects specialized models per task. Build a clean abstraction layer that lets you swap the underlying model without touching business logic. You'll need it again in 6 months.

   Local Inference Became Practical

   PFlash speculative prefill hits 10x over llama.cpp at 128K context on an RTX 3090, using a Qwen3-0.6B drafter. Qwen 3.6 35B-A3B runs on an AMD 7700 XT at 128K with flash attention. The floor for self-hosted, cache-optimized agent infrastructure dropped this week.

   There is a catch that several sources flagged independently. Open-weight models benchmark well and underperform in agent workflows. The reason is mechanical: LangChain, OpenCode, and similar harnesses are tuned to proprietary API conventions — tool-call format, system prompt layout, retry behavior. Swap in a local model and the scaffolding is wrong. The fix is not a better model. The fix is a model-specific harness. Log tool-call traces from the API run and the local run, diff them, build the adapter.

   ---

   Platform Design Shifts

   Hugging Face is redesigning for agents as the primary consumer by end of 2026. Three patterns worth adopting now: agents.md at repo roots for machine-readable project context, token-efficient API responses with small payloads and short stable field names, and headless-first interfaces where the CLI is canonical and the UI is a view. This is not vision. It is shipping code from a team with 15M users and 200 engineers.

   Action items

   • Audit all Codex API integrations and migrate to GPT-5.5 endpoints before OpenAI deprecation
   • Add agents.md files to your top 5 public-facing repos this sprint — describe entry points, test commands, and sharp edges in machine-readable format
   • Prototype a hybrid local+API inference architecture: use a small-active MoE (Qwen 3.6 35B-A3B or similar) for triage/pre-screening, route only complex tasks to frontier APIs
   • Build model-specific agent scaffolding before concluding local models are worse — log tool-call traces and diff API vs. local runs

   Sources:On our agent workload, KV cache residency dominated cost · The next caller hitting a given API is not going to be a person with a browser · Your AI model abstraction layer just became critical

3. 03

   The Planner/Executor Split — Hard Isolation for Agent Security

   The Pattern

   The new agent security primitive worth naming this week is the Planner/Executor Split. Two LLM instances. A hard privilege boundary between them. The planner sees the tools and the plan, never untrusted input. The executor sees untrusted content, never the tools. Prompt injection lands on the component that has no authority to act.

   This isn't a prompt engineering trick. It's privilege separation ported to LLMs, the same mechanism Unix uses to keep user processes out of root. Gmail already runs stacked defenses on this pattern in production.

   Skip the planner/executor split and every tool call is one clever paragraph away from exfiltration.

   MCP vs Skills: A Security Boundary Decision

   MCP-versus-Skills gets pitched as a bake-off. Read the capability surface and it's a security boundary decision. The asymmetry that matters:

   Dimension	MCP	Skills
   Isolation	Process-level (JSON-RPC over separate container)	None (runs in agent's environment)
   Execution	Typed parameters, schema validation	Arbitrary bash, python, curl
   Versioning	Server redeploy required	File change
   Attack surface	Bounded by schema	Unbounded if influenced by untrusted input

   A Skill that ingests untrusted content and shells out is a prompt injection to RCE chain. Not theoretical. The split I'd ship: MCP for anything touching production. Skills for dev-facing agents in CI runners, ephemeral containers, and workstations. Nowhere else.

   The Defense Stack

   The complete taxonomy has two layers. One makes injection harder. The other caps what happens when it lands.

   Model-level (probabilistic — makes injection harder)

   • Spotlighting: wrap untrusted text in control tags (<UNTRUSTED>) and tell the system prompt to treat tagged content as data only
   • Instruction Hierarchy: fine-tune the model to rank system prompts above user messages above third-party content

   System-level (deterministic — caps blast radius)

   • Planner/Executor Split: hard privilege boundary, 2x inference cost
   • Least-Privilege MCP: minimum operation set per server, typed schemas that reject unexpected params
   • Human-in-the-Loop: checkpoint gates on irreversible actions

   The Planner/Executor Split is not free. 2x inference, coordination latency, and a handoff protocol you have to design and maintain. For any agent that reads external content and takes write actions, it's the only pattern that stops a single injection from doing damage. Pay the tax.

   Action items

   • Audit your agent architecture against the MCP/Skills taxonomy this sprint — map every integration: live system → MCP, procedural knowledge → Skills. Flag any Skills executing in unsandboxed environments as P1
   • Implement Spotlighting (<UNTRUSTED> tags) on all LLM calls processing external content within 2 weeks
   • Prototype the Planner/Executor Split for your highest-risk agent workflow — any agent that reads untrusted content AND has write tool access
   • Apply least-privilege scoping to all MCP tool definitions — each server exposes minimum operations with typed parameter schemas that reject unexpected input

   Sources:MCP and Skills solve different problems · The next caller hitting a given API is not going to be a person with a browser