Page 12
12 articlesYour Eval Suite Is the Product Spec You Refused to Write
Fuzzy PRD adjectives like 'helpful' and 'concise' don't survive contact with a model — the eval suite is where those decisions actually get made. Treat the eval as the spec, not as instrumentation.
Fallback Path Atrophy: Your Graceful Degradation Stopped Working Three Months Ago
The fallback you wrote nine months ago is silently broken. How AI graceful-degradation paths bit-rot, why integration tests miss it, and the failure-injection discipline that keeps degraded mode shippable.
Forced Conformance Bias: When the Model Rounds Your Intent to the Distribution Mode
Aligned LLMs quietly round unusual requests toward the training distribution mode. Here is why standard evals miss it and the off-mode discipline that catches it.
The Frozen Prompt: When Your Team Is Afraid to Edit a System Prompt That Works
A 4,000-token system prompt nobody dares edit is not stability, it's debt. How prompts freeze, why iteration collapses around them, and the archaeology and eval discipline that thaws them.
Hidden SDK Retries: Why You're Paying Twice and Don't Know It
Major LLM SDKs ship with two automatic retries by default. Stack a caller-side retry on top and a single request fans out to nine inference calls during a provider blip — invisible in your traces, visible only on the bill.
The Internal-Tooling Agent: When Your Highest-Leverage AI Feature Has Zero Customers
Customer-facing AI features absorb most of the budget, but the highest-leverage AI investment in your company is the internal Slack bot nobody is staffing. Here's the math, the failure pattern, and the discipline that captures the value.
Negative Prompts Are Code Smells: Why Every 'Don't' in Your System Prompt Is Technical Debt
Every 'do not' clause in a production system prompt is a patch on a behavioral mismatch. Track negative-prompt density, refactor each negative into a positive specification, and use the residue as a signal that prompt engineering is the wrong tool for the job.
OAuth in MCP: Threading User Identity Through Tool Servers
MCP standardized how agents get tokens for tool servers, but left the harder question — how those servers thread user identity to downstream APIs — to implementers. Here is what holds up under audit.
The Phantom Skill: When Your Agent Demonstrates Capabilities You Never Tested For
When users build workflows on AI agent behaviors no test ever verified, you're shipping capabilities you cannot defend. A discipline for finding phantom skills before the next model upgrade silently removes them.
The Policy File: Why Your Refusal Rules Don't Belong in Your System Prompt
Production system prompts are three config files in a trench coat — conversational voice, output formatting, and refusal policy crammed into one artifact with one reviewer and one release cadence. Every policy edit becomes a behavioral regression on unrelated tasks. Here is the factoring that pays for itself.
Production Bias Auditing: Catching AI Discrimination Before Your Users Do
Pre-launch fairness audits expire the moment a model meets real traffic. A practical playbook for the metrics, slice-level audits, regression gates, and monitoring infrastructure that catch AI bias drift before it reaches users.
Prompt Edits Aren't Wording Changes: A Code Review Discipline for Prompts as Software
Prompt edits look like English but behave like code. The review discipline — paired eval-and-prompt PRs, behavioral diff comments, split reviewer roles — that catches behavioral regressions before users do.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
