Page 27
12 articlesDogfooding Is Not an Eval Strategy
Your team using the product every day is a smoke test, not an eval. Why builders are the worst sample of their own users, and how to measure AI product quality on the traffic that actually breaks it.
The Embedding Upgrade That Silently Re-Ranks Your Entire Corpus
Swapping your embedding model for a higher-benchmarking one invalidates every vector you have stored. Why the upgrade silently degrades retrieval, and how to migrate it like a schema change.
The Eval Set That Got Easier While You Weren't Looking
An LLM eval suite that everything passes has stopped measuring anything. Why static eval sets saturate, how to spot it, and how to keep a usable score gradient.
The Eval Set Is a Lagging Indicator: Your Green Dashboard Only Knows Last Quarter's Failures
An eval suite grown only from postmortems certifies your AI system against the past. Here is why a green pass rate lies on migration day, and how to fund exploratory coverage.
Your Eval Set Is a Frozen Photograph of Traffic Your Users Already Left
A benchmark gain measures progress on a distribution your users already left. How eval-set staleness, the survivorship trap, and a single aggregate score hide a silent decline — and how to keep your eval tracking the river.
The Eval Suite That Became the Spec Nobody Agreed To
Most agent teams have no requirements doc — the eval suite became the spec by default. Why a green eval run certifies one engineer's assumptions, and how to give the eval set the review rigor of an API schema.
The Fallback Model You Never Load-Tested
A configured fallback proves your router works — it proves nothing about whether your application survives the secondary model's output. Why named fallbacks fail under real traffic, and how to test the failover before your provider does it for you.
Your Fallback Path Is the Only Untested Code in Production
The degraded path you built to survive an outage runs almost never, so it rots quietly and makes its debut during the exact incident it was designed to survive.
The Happy Path Is the Only Path Your Agent Eval Ever Tested
A 94% pass rate measures how good you are at imagining success, not whether the agent works. How golden-path bias creeps into agent eval suites and how to fix it with failure-mode coverage, harvested production cases, and fault injection.
The Idempotency Key Your Agent Never Sent
When an agent's tool call times out, it retries — and without an idempotency key, that retry charges the card again. How to make agent retries harmless instead of dangerous.
The Incident Ticket With No Repro Steps: Reproducibility as Something You Engineer
An agent deleted the wrong record and the postmortem has a hole where the cause should be. Reproducibility for AI agents is not inherited from your stack — it is something you capture, version, and replay on purpose.
Your Internal API Became a Public API the Day an Agent Called It
An internal API stays internal only as long as you can name every caller. Wire an LLM agent to it and the contract you never wrote down becomes a liability — here's the public-API discipline you suddenly owe it.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
