CacheEconomicsandDomainRoutingReshapeModelSelection

◆ DEEP DIVES

1. 01

   Cache Hit Rate Is the New Eval Metric — and It's a Bigger Cost Lever Than Model Quality

   The Shift Nobody Priced In

   Agentic workloads spend most of their tokens inside loops: retries, tool calls, multi-turn reasoning against stable system prompts. Repetitive prefixes are exactly what KV cache reuse exploits. DeepSeek V4 Pro ships a disk-backed KV cache that persists for hours, versus the roughly 5-minute TTL that is the industry standard. One shared production dashboard reported $3,351 in cache savings against $1,051 in API spend. That is a 3.2× effective discount, and it does not show up on a price-per-token comparison sheet.

   This is not really a DeepSeek story. It is a serving-architecture story. V4 Pro's hybrid CSA/HCA attention compresses KV cache to 10% of standard size and reports ~4× lower long-context FLOPs. At the concurrency and context lengths a real coding agent harness produces, effective cost between models swings 2–4× depending on serving stack. That delta is larger than the quality gap between open and closed models on most coding tasks, which is the comparison most teams are actually running.

   ---

   The Pricing Floor Has a Hidden Fee

   Grok 4.3 sets the headline floor at $1.25/M input, $2.50/M output, 40–60% below Grok 4.2. xAI also introduced a $0.05 fee per safety-filter-blocked request. At a 2–3% filter rate on production prompts, that erodes token savings meaningfully at scale. Most cost dashboards won't catch it because they track tokens, not rejections.

   A model that is cheap to serve is not the same as a model that is cheap to trust. The blended cost on a replay of real traffic is the number that decides the migration, not the sticker price.

   Cross-Source Pattern

   Four independent sources converge on the same read: inference economics, not benchmark scores, are the binding constraint on production model selection. Nebius paid $615M for Eigen AI specifically for inference optimization. Cursor's −23% gross margins show what a thin wrapper over expensive inference looks like on the P&L. Grok's own commentary notes the headline price cut may be subsidized by poor utilization and is unlikely to beat a well-cached DeepSeek workload.

   What To Do

   The immediate action is observability, not migration. Most agent runtimes emit tokens/sec and $/query but not cache-hit rate, prefix-reuse ratio, or effective $/1K tokens net of cache discounts. Those three metrics are now first-class cost variables. Stable system prompts and tool schemas are where the DeepSeek-style discount actually lives, which makes harness design a cost lever, not an infrastructure footnote.

   Action items

   • Instrument cache-hit rate, prefix-reuse ratio, and effective $/1K tokens (net of cache) as first-class metrics in your agent runtime by end of sprint.
   • Replay last month's agent traffic through DeepSeek V4 Pro and Grok 4.3 in shadow mode; compare blended $/successful-task, not sticker $/M tokens.
   • Add refusal-rate instrumentation to your LLM gateway and model the $0.05/blocked-request fee into unit economics.
   • Evaluate whether your system-prompt and tool-schema structure maximizes prefix reuse; refactor for cache efficiency before switching models.

   Sources:AINews · Techpresso · StrictlyVC · Turing Post

2. 02

   Your Benchmarks Are Measuring Harness Fit — Fix the Eval Before Fixing the Stack

   The Evidence Harness-Dependence Is Load-Bearing

   Cross-vendor benchmark comparisons are not just noisy. They are systematically confounded by the evaluation harness, and this week produced the receipts:

   1. GPT-5.5 beats Opus 4.7 overall on the Intelligence Index but loses on PostTrainBench when evaluated inside the Claude Code harness. Same weights, different plumbing.
   2. Grok 4.3 gained +321 Elo on GDPval-AA yet regressed on Vending-Bench 2 so badly that the agent reportedly preferred to 'sleep' rather than act. The same model on a different eval surface produces the opposite conclusion.
   3. Grok 4.3 ranks #1 on CaseLaw and CorpFin but scores 11% on ProofBench. Within-model domain variance is wider than the between-vendor gap on any single domain.

   Hugging Face's Clem Delangue frames this as the 'engine vs car' problem: closed APIs bundle routing, tool schemas, retry logic, and sometimes multiple models behind one URL. Benchmarking an open-weight model through a harness designed for Claude's tool-use schema produces degradation that looks like a model problem but is an instrumentation problem.

   ---

   The Qwen Signal

   Alibaba's Qwen3.6-27B reportedly beats a predecessor 15× its size on coding benchmarks. The thing this headline doesn't tell you is whether contamination was ablated (it wasn't disclosed), and coding benchmarks reward single-file completions rather than the multi-repo edits where production assistance actually fails. The cheapest experiment is an A/B on held-out pull requests, not a re-run of HumanEval.

   Cross-harness comparisons correlate with production outcomes. They do not cause them, and they do not predict them well enough to justify a migration on their own.

   The Fix

   The eval harness itself needs to become a logged, versioned variable. Two changes make this concrete:

   • Log harness config as an eval dimension. Prompt format, tool schema shape, retry policy, context budget all differ across model families and all affect measured quality. Re-test top open candidates (V4 Pro, Kimi K2.6, Qwen3.6-27B) with per-family tuned harnesses.
   • Require vendors to publish harness methodology. Any procurement checklist that accepts a benchmark score without the harness definition, tool-call budget, and retry policy is accepting an unauditable claim.

   The open-weight gap is now 5 points on the Intelligence Index (52–54 vs. 57–60), concentrated in HLE, CritPt, TerminalBench Hard, and hallucination-heavy Omniscience, rather than general coding or agentic tool use. On multi-turn agentic coding specifically, V4 Pro is called the first open-weight model that genuinely feels comparable to Codex or Claude Code. If internal evals confirm a gap of ≤4–5 points on task-completion metrics, a two-week shadow-traffic bake-off is warranted. If the gap is larger, re-baseline the evals first.

   Action items

   • Re-run your top-3 agent evals on DeepSeek V4 Pro, Kimi K2.6, and Qwen3.6-27B with per-model-family harness tuning (prompt format, tool schema, retry logic) this sprint.
   • Add harness config (prompt format, tool budget, retry policy) as a required metadata field in all eval reports.
   • A/B test Qwen3.6-27B against incumbent on held-out internal pull requests, measuring pass@1, latency, and $/1K tokens self-hosted vs. API.
   • Build a per-domain benchmark slice dashboard showing model variance across task types before any migration decision.

   Sources:AINews · Techpresso · Turing Post · THE DECODER

3. 03

   Agent Defense Patterns Harden: Planner/Executor Split, MCP vs SKILL.md, and the Security Surface Nobody Sandboxed

   The Reference Architecture for Untrusted Content

   The Planner/Executor Split is settling in as the default defense pattern for agents that touch untrusted content: emails, web pages, user uploads, RAG retrievals. The shape is simple. A planner LLM has tool access but never sees untrusted text. An executor LLM reads untrusted text but has no tools. Gmail is cited as a production reference. The honest cost is that it roughly doubles inference spend. That is the price that turns indirect prompt injection from catastrophic into recoverable.

   The thing this doesn't solve is exfiltration through the planner's own tool calls. The eval you actually want is not "did the executor get fooled." It is "did a document change the planner's plan in a way the user didn't ask for." Build that eval before the architecture, or the architecture ships without a way to measure whether it works.

   ---

   MCP vs SKILL.md: A Systems Decision, Not a Taste Decision

   The framing that MCP and Skills are competing approaches is wrong. They are orthogonal primitives:

   Dimension	MCP	SKILL.md
   Purpose	Integration plane (live systems, state, auth)	Knowledge plane (procedures, playbooks)
   Runtime	Separate process, containerized	Agent's own environment
   Invocation	Typed JSON-RPC, schema-validated	Agent reads markdown, runs bash/python/curl
   Primary risk	Infra overhead, auth sprawl	Arbitrary code execution with no isolation

   The security surface most teams miss: Skills execute arbitrary bash/python/curl in the agent's own environment with no sandboxing. This is remote code execution by design. Every SKILL.md file deserves the same review rigor as a Dockerfile. Without Firecracker, gVisor, or at minimum a seccomp-restricted subprocess, a compromised Skill is a privilege-escalation vector.

   The cost asymmetry: an MCP server that should have been a SKILL.md costs you a container, a deploy pipeline, and an on-call rotation. A SKILL.md that should have been MCP costs you correctness on a slice of traffic. One is a line item. The other is a bug.

   Stacking Defenses

   OWASP ranks prompt injection as the #1 LLM threat, and the guidance is explicit: no single fix exists. The layered approach combines:

   • Spotlighting: Wrap untrusted text in <UNTRUSTED> tags with a system-prompt rule. Hours of work, meaningful floor raise.
   • Instruction Hierarchy: Fine-tune to rank system > user > third-party content.
   • Planner/Executor Split: Architectural isolation for high-stakes paths.
   • Least-Privilege Tools: Minimize blast radius per tool call.

   Spotlighting is the cheap floor and worth taking today. Planner/Executor earns its doubled inference cost on any path that touches the open web, and less obviously elsewhere. Skill sandboxing is the one that teams defer and regret; a compromised SKILL.md in production is not a bug you recover from quickly.

   Action items

   • Implement Planner/Executor Split for any agent ingesting untrusted content (emails, web pages, user uploads, RAG retrievals) this quarter.
   • Ship Spotlighting (<UNTRUSTED> wrapper + system-prompt rule) on every RAG and tool-output path by end of week.
   • Audit all agent tools: classify as MCP-worthy (live state, auth) or Skill-worthy (procedural). Kill any MCP server that is wrapping a prompt template.
   • Sandbox all SKILL.md execution in Firecracker, gVisor, or seccomp-restricted subprocess before any Skill reaches production.

   Sources:ByteByteGo · AINews