Page 4
12 articlesThe Hallucinated Tool Argument That Passed Schema Validation
JSON schema only validates shape, not truth. When agents hallucinate referential arguments that pass schema checks, retry loops launder the bug into a clean audit trail — here is the missing layer.
The Heavy Tail Your Token Forecast Never Priced
Pilot-based token cost forecasts miss the heavy tail of production users — and the bill arrives at the p99, not the median. How to price the distribution instead of the mean.
The Inference Region Your Data Residency Policy Forgot to Pin
Provider region parameters look like AWS region pinning but behave like routing hints. Engineering teams that confuse the two ship a residency posture that only survives until the first real audit.
The Interrupt UI That Taught Your Users to Never Interrupt the Agent
A streaming agent's stop button can train users to wait through bad answers instead of redirecting them. The fix is to treat interrupt as a turn in the conversation, not a circuit breaker on the API call.
The Kill Switch With a Latency Budget Your Incident Never Met
An AI feature whose containment time exceeds its blast time has shipped a kill switch on paper, not in practice. Measure activation latency, tier it against damage rate, and write the number in the runbook.
The Latency-Budget Router That Was a Quality-Loss Router by Another Name
A latency-budget router does exactly what its loss function asks for and silently downgrades the reasoning-eligible cohort. Why the aggregate eval hides the regression and what to instrument instead.
The Legal Review Timeline Your AI Feature Roadmap Never Costed
Legal review is a serial dependency on a parallel roadmap. The team that learns this on the first launch slip pays for it in every quarter after.
The Localized System Prompt Your Model Performs Worse Against Than the English Original
Translating a tuned English system prompt into 14 locales is not localization — it is a silent eval regression nobody re-measures. The model's instruction-following accuracy drops 8–22 points and your non-English users get an agent that ignores the constraints the English users see honored.
The Middle-Context Blindness Your Retrieval Pipeline Never Measured
Retrieval@10 stays green while answer quality drifts. The gap is a U-shaped attention bias that lives in the seam between the retrieval team and the prompt team, and neither dashboard knows the slot the model never read.
The Model Card Your Procurement Team Treated Like a Datasheet
Procurement reviewers read model cards as contractual representations, not research disclosures. Author a separate vendor due-diligence package before legal binds you to claims your engineering team wrote as narrative.
The Model Deprecation Notice That Landed During Your Code Freeze
Provider deprecation cadence is not external weather. Treat vendor clocks as production infrastructure so the next sunset notice doesn't re-prioritize your quarter.
The OAuth Scope Your Agent Acquired Across Chained Tool Calls
OAuth consent treats agents like single-purpose apps, but each chained tool call expands the realized authority your audit log has to explain. The one-shot screen rendered the worst case as one decision.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
