Page 4
12 articlesYour Model's Confidence Score Is a Vibe, Not a Probability
An LLM's self-reported confidence isn't a probability — it's a fluent token shaped by RLHF incentives. Why calibration is the property you never measured, and what to gate on instead.
Your Eval Suite Is Overfit to Your Incumbent Model
A private eval suite grows from one model's production failures, so it flatters the incumbent and slanders every challenger. Separate regression cases from capability cases and test on the traffic you actually serve.
Acceptance Sampling for Agent Output: What Manufacturing QA Knows That Code Review Doesn't
Agents ship hundreds of PRs a week and most get no review at all. Manufacturing solved this problem a century ago with acceptance sampling — AQL tables, lot rejection, and switching rules that turn agent trust into a stated, measurable policy instead of vibes.
Be Strict in What Your Tools Accept: Postel's Law Fails Agent Systems
Liberal coercion of malformed tool arguments feels robust but silently converts schema violations into data corruption your evals never see. Strict rejection with model-legible errors is what makes an agent loop self-correcting.
Best-of-N Is an Architecture, Not a Benchmark Trick
Running N parallel attempts behind one answer often beats one shot from a bigger model — if you engineer the judge, decorrelate the failures, and budget N by stakes. The cost math, the selector's bias problem, and the confidently wrong survivor.
Blast Radius Is the Permission Model: Sandbox Agents by What They Can Break, Not What They Can Read
Every command the Replit agent ran before deleting a production database was authorized — the permission model answered the wrong question. Why agent safety lives in the execution environment: scoped tokens, ephemeral branches, egress allowlists, spend caps, and approval gates placed at blast-radius boundaries.
CODEOWNERS for the Fleet: Review Routing When the Author Is an Agent
Path-based CODEOWNERS assumes the author knows whose turf they're on — AI agents don't. Why agent diffs deadlock review queues, and how sponsor-based routing, per-directory autonomy budgets, and pattern-plus-sampling codemod review fix it.
Deprecating an API When Your Biggest Client Is a Prompt
AI agents don't read changelogs — their knowledge of your API is frozen in prompts, tool schemas, and training data. Why turning off v1 causes retry storms instead of migrations, and how adapters, sunset headers, and agent-readable errors fix it.
Dogfooding Your Agent Is Not QA
Internal users silently fix your agent's mistakes, dodge its weak spots, and trade workarounds in Slack — so dogfood metrics read clean until real customers churn. Instrument repair events, edit distance, and cold-user cohorts instead.
FinOps for Tokens: Attributing AI Spend to the Feature That Caused It
Your LLM invoice can't name the feature that spent the money. Why prompt caches, batch APIs, and multi-tenant agents break cost attribution — and the tagging, span-level metering, and showback discipline that fix it.
Flaky Tests Poison Agent Loops Faster Than They Ever Hurt Humans
A human shrugs at a flaky test; an agent takes the red as ground truth — reverting correct changes and burning tokens on phantoms. Why test determinism is now an SLO, and how agent fleets detect and fix the flakes they suffer from.
Game Days for Agents: Rehearsing the Failure You Can't Reproduce
Chaos engineering assumes replayable failures; agent systems fail through stochastic paths that never recur. Game days — tool-fault injection, degraded-model drills, poisoned-context scenarios, and escalation fire drills — build the operator muscle memory and expose the trace-tooling gaps that replay never will.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
