Page 8
12 articlesThe Scaffolding Audit: Every Model Release Turns Part of Your Harness Into Dead Weight
Retry choreography, JSON repair parsers, and forced chain-of-thought were built for a model you no longer run. A practical discipline for tagging, ablating, and deleting obsolete LLM scaffolding on every model upgrade.
The Single Quality Number That Doesn't Exist
Your AI feature's quality is a distribution, not a scalar. Why averaged eval scores report segment collapses as improvements, what to put on the executive slide instead — segment grids, floor metrics, worst-case transcripts — and the reporting cadence that keeps a noisy metric credible.
The Telephone Game in Your Agent Pipeline
Every handoff in a multi-agent system is lossy compression nobody tuned: constraints die, qualifiers vanish, and errors compound across hops. How to measure information loss per handoff, decide what travels verbatim, and fix it with schemas instead of prompts.
The Ten-Thousand-File Codemod: Running an Agent Fleet Over a Mechanical Migration
Deterministic codemods convert the easy 80% of a migration and stall on the long tail — exactly where per-file coding agents shine. But at fleet scale the problem stops being prompting and becomes batch operations: sharding, verification gates, quarantine, and merge strategy for a MapReduce job whose mapper is stochastic.
The Thousand-Agent Customer: Per-Seat SaaS Wasn't Priced for Machines
AI agent fleets break seat licenses, automation clauses, rate limits, and per-user analytics all at once. Why vendors are splitting into agent-hostile and agent-priced camps — and the five clauses to read before wiring an agent to any SaaS contract.
The Thundering Herd Behind Your 429s: Rate Limits Are a Distributed Systems Problem
Exponential backoff without jitter organizes your retry herd instead of dissolving it — synchronized 429 waves, retry amplification, and metastable outages follow. Why LLM rate limits demand jitter, retry budgets, and the client-side scheduler you didn't plan to build.
The Tokenizer Tax: Your AI Feature Costs More and Works Worse in Every Language but English
The same AI feature costs 2-3x more to serve in Japanese or Arabic than in English, and delivers measurably worse quality — yet most teams ship globally with a single English eval suite. How token fertility drives per-locale cost, context, and accuracy, and what to measure before a global launch.
The Transcript Remembers What the Commit Message Forgot
Every agent session logs the rejected approaches, discovered constraints, and reasoning that commit messages compress away. How to mine transcripts into searchable institutional memory — and handle the surveillance tension that comes with it.
The Warm Sandbox Pool: Infrastructure Economics When Every Agent Task Gets Its Own Machine
Per-task agent isolation quietly turns your platform into an ephemeral-VM fleet. The real engineering lives in warm pools, snapshot pipelines, duration-aware bin-packing, and reaping abandoned sandboxes — and your container instincts will mislead you.
The World Has No Staging Environment
Agents that act on Salesforce, QuickBooks, or Gmail run against production because no other tier exists. Why dry-run flags are theater, where vendor sandboxes break down, and how to build a real rehearsal tier with replay proxies and synthetic tenants.
Two Writers, One Working Tree: Concurrency Control for Human-Agent Co-Editing
Coding agents editing your checkout while you type are a concurrency bug, not an AI quirk. Lock, isolate, or merge — the database playbook applied to human-agent co-editing.
Wardley-Map Your AI Stack Before You Build Another Layer of It
Build-vs-buy debates for AI infrastructure assume a stable landscape that doesn't exist. A Wardley map shows which layers of your stack are commoditizing in months — and which components, like domain evals and proprietary data, move the other way.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
