HarnessEngineeringBeatsModelSwapsonEvalGains

◆ DEEP DIVES

1. 01

   Harness Engineering Is Your New Competitive Moat — Not Model Selection

   The Data Is In: Orchestration Beats Intelligence

   The traditional assumption — 'wait for the next model upgrade to hit our quality bar' — is now empirically wrong. Multiple independent benchmarks released this week confirm that the harness (prompt design, tool definitions, error recovery, middleware) is a larger lever than model choice for AI feature quality.

   Agentic Harness Engineering improved Terminal-Bench 2 pass@1 from 69.7% to 77.0%, beating a human-designed Codex-CLI baseline at 71.9% — and it transferred across model families while reducing token use by 12%.

   HALO showed even more dramatic results: AppWorld improvement from 73.7 to 89.5 on Sonnet 4.6 through recursive self-improvement of the harness, not the model. These aren't marginal gains — they're the difference between a demo and a production feature.

   ---

   The Cost Dimension Makes This Urgent

   IBM's Granite 4.1 8B used only 4M output tokens on the Artificial Analysis Intelligence Index versus 78M for Qwen3.5 9B — a 19.5x efficiency gap that's entirely about harness and architecture optimization. Shopify fine-tuned a smaller open-source model on Flow-specific data and achieved higher accuracy, lower latency, AND lower cost than general-purpose frontier models. When your inference costs are growing 10-15x in six months, a 19.5x efficiency difference isn't a nice-to-have — it's the difference between a viable product and a margin crisis.

   Commercial Infrastructure Is Emerging

   This isn't just research anymore. LangChain launched Harness Profiles — versioning per-model prompts, tools, and middleware configurations. DeepAgents Deploy shipped low-code agent deployment with LangSmith tracing. The pattern is clear: harness quality is becoming a managed, versionable, evaluable artifact alongside your code.

   What This Means For Your Team

   • Model selection is now a product decision, not an engineering default. Features where errors cost minutes belong on cheaper models with optimized harnesses. Features where errors cost hours earn frontier models.
   • Your harness — the prompts, tool definitions, error recovery patterns — should be versioned like code and evaluated per model family.
   • Netflix's LLM-as-a-Judge blueprint (600 golden examples, 83-92% accuracy) provides the evaluation framework to measure harness improvements objectively.

   Action items

   • Create a harness versioning and evaluation pipeline for your top 3 AI features this sprint
   • Run a model-tier routing analysis: classify every AI feature as 'errors cost minutes' vs 'errors cost hours' and assign model tiers accordingly
   • Build 600 expert-labeled golden examples for your primary AI feature following Netflix's blueprint
   • Evaluate Shopify's approach: benchmark a fine-tuned small model against your current frontier API for your most well-defined use case

   Sources:Your AI cost model is about to break · An agent opened a checkout flow at 3:14 in the morning · A product manager at a mid-size SaaS company looked at her inference bill · Shopify proves fine-tuned small models beat GPT-class LLMs · A head of product opened her cloud bill last month · A product manager on an AI team opened her evaluation dashboard

2. 02

   The Token Cost Reckoning: 15x Growth, Zero Discounts, and the Vendor Lock You Didn't Price

   The Numbers From 15 Companies Are Worse Than Expected

   A staff engineer at a late-stage fintech watched one of her developers burn through a day of Claude Code and realized the bill looked like a second salary. That is the texture behind Gergely Orosz's survey of 15 companies, which is the most granular AI cost data available this quarter. Token spend grew 10-15x in six months at both a large enterprise and a seed-stage startup. Individual developers are spending $500/day on Claude Code, which staff engineers say has 'practically doubled' employee costs. One seed-stage company went from $200/dev/month to $3,000/dev/month.

   About half of surveyed companies are on the 'let it rip' plan, which is a pitch-deck word for unforecasted cost. The Series D fintech that called its token spend 'unsustainable' at an all-hands is the preview.

   Anthropic's Extraction Era Is Real

   Anthropic offers zero enterprise discounts even at $5M+/year spend. Teams describe silent Claude Code nerfing, outright bans, and aggressive price increases, while nearly every company in the survey is Claude-dependent. What customers actually bought is not the API call. They bought every prompt, eval, and guardrail tuned against one model family, and that is the switching cost that keeps them paying.

   The Cost Lives in the Wrong Budget Line

   The PM decision this quarter is a budget-line decision, not a model decision: do tokens belong in COGS or R&D? If tokens sit in COGS, the product team owns unit economics and a 3x reduction in retrieval spend becomes a roadmap item with an owner. If tokens sit in R&D, nobody owns it and the bill keeps growing. One RAG app consumed 10.4M tokens via naive Supabase MCP integration. The same job ran on optimized infrastructure for 3.7M tokens. That is a 64% reduction from an architecture choice, not a model change.

   ---

   Open-Weight Models Are the Pressure Valve

   Open-weight pricing has collapsed to $1-3/M output tokens (Qwen 3.5 Plus at $3/M, MiMo-V2.5 Pro at $1/$3/M). LangChain staff explicitly state closed models are 'too expensive for many agent workloads.' Features that were economically impossible at $15-60/M tokens become viable at these prices. Three Chinese labs dropped frontier models in a single week: Alibaba's Qwen 3.6 Max, Moonshot's 1T-parameter Kimi-K2.6, and MiniMax M2.7 (fully open-source, 56% SWE-Pro).

   Provider	Output Token Cost	Enterprise Discount
   Anthropic (Claude)	$15-60/M	None at $5M+/yr
   Qwen 3.5 Plus	$3/M	Open-weight
   MiMo-V2.5 Pro	$1-3/M	Open-weight
   MiniMax M2.7	Free (OSS)	N/A

   Action items

   • Audit every AI-dependent feature for cost model accuracy this sprint — re-forecast COGS using current token pricing at 10x whatever was modeled 6 months ago
   • Map Anthropic dependency across customer-facing features AND internal dev tooling — identify which capabilities break if prices rise 2-3x
   • Implement per-feature token budgets with hard circuit breakers and real-time alerting before end of quarter
   • Benchmark open-weight models (Qwen 3.5, MiniMax M2.7, Kimi-K2.6) against your current provider for your top 3 AI features

   Sources:A product manager at a mid-size SaaS company looked at her inference bill · Your AI cost model is about to break · A head of product opened her cloud bill last month · Agents just hit production-grade benchmarks · A product lead at a Series B company opened the OpenAI status page

3. 03

   Your AI Feature Stack Is a Security Liability — 8+ Critical RCEs This Week Alone

   AI Tooling Ships With 2005-Era Security

   A security engineer opened her advisory feed on a Tuesday and counted eight AI-adjacent tools with CVSS 9.8-10.0 vulnerabilities disclosed in a single week. Claude Code hit 10.0 for a symlink sandbox escape. Cursor's AI agent allowed arbitrary code execution via malicious git clone. GitHub Enterprise Server exposed millions of repositories via crafted push requests. None of these are novel vulnerability classes. They are hygiene failures shipped at startup speed.

   The evaluation question for an AI framework is not which one has the best features. It is which one will not be a 10.0 headline next week.

   The Weaponization Window Collapsed

   Sysdig observed attacks on LMDeploy within 12.5 hours of disclosure, without a public proof-of-concept, because advisory text alone was enough for an LLM to generate the exploit. MOAK autonomously exploits 98% of Known Exploited Vulnerabilities using publicly available models. Disclosure-to-exploit is measured in hours now. A patching SLA written against a weekly cadence is a patching SLA for a threat model that no longer exists.

   ---

   Supply Chain Attacks Are Industrializing

   73 malicious GlassWorm extensions appeared in Open VSX in April 2026 alone. The TeamPCP/UNC6780 campaign hit npm, PyPI, and Docker Hub within 48 hours. Dependabot carried the attack through Bitwarden CLI's merge gates. The automation did exactly what it was configured to do, which is the uncomfortable part. LiteLLM's SQL injection exposed every downstream API key (OpenAI, Anthropic, Bedrock) via pre-auth attack, exploited within 36 hours.

   The PM Responsibility

   Separate the thing being pitched from the thing being done. The pitch is "adopt AI tools to move faster." The thing being done is adding a new attack surface with its own incident timeline. 88% of GitHub Enterprise Server instances remain unpatched, and DPRK's HexagonalRodent is using Cursor and ChatGPT to vibe-code malware loaders. The forcing function for this sprint: no AI tool enters the stack without a named owner, a patch SLA measured in hours, and a documented kill switch. Teams that write that policy this sprint do not run the fire drill next quarter.

   • Claude Code: CVSS 10.0 (sandbox escape, patched v2.1.64)
   • Cursor: RCE via git clone (patched v2.5)
   • GitHub Enterprise: RCE via push options (88% unpatched)
   • LiteLLM: Pre-auth SQL injection exposing all API keys
   • Open VSX: 73 malicious extensions in one month

   Action items

   • Verify Cursor is updated to v2.5+ and GitHub Enterprise Server is patched across all environments — escalate to platform team within 24 hours
   • Add mandatory security review gate to your AI tooling evaluation process — no AI framework enters your stack without a vulnerability assessment
   • Audit your LLM proxy/gateway layer for credential isolation — specifically verify that provider API keys are not stored in a single queryable database
   • Implement dependency cooldowns across your CI/CD pipeline and pin container images by digest, not tag

   Sources:A security engineer opened the vulnerability dashboard · A developer opened her IDE on Monday morning · A developer on your team opened the dependency graph · A product manager on call this week got paged at 2am · A security engineer on your team opened the weekly CVE digest · A platform trust lead opened her dashboard