Page 15
12 articlesA Prompt Diff Hides Its Own Blast Radius
A three-word prompt edit and a three-paragraph rewrite look identical in a text diff, yet their behavioral consequences are not. Why prompt review needs eval deltas, not character counts.
Re-Ask Rate: The Failure Signal Your Eval Pipeline Never Extracts
Users who repeat the same question in a session are telling you the previous answer failed — but turn-level evals and end-of-session CSAT both miss it. How to instrument re-ask rate as a first-class metric.
Shadow Replay Punishes the Model That Would Have Changed the Conversation
Shadow replay evaluations quietly punish better models by scoring them against logged user turns that were conditioned on the old model — here is why, and what replay can still measure honestly.
Streamed Tokens Are a Promise You Can't Take Back
A confident token-streamed answer collapses when its backing tool call fails. Streaming is an irreversibility contract — and there are patterns that buy back optionality without giving up perceived latency.
The Agent That Read Last Week's Slack Like It Was Yesterday
An agent retrieves a six-week-old message that says 'we'll ship tomorrow' and treats it as a present-tense plan. The retrieval pipeline kept the body and threw away the clock.
The Agent That Wouldn't Stop: Scope Creep as a Runtime Failure Mode
Agents fix the bug, then keep going — refactoring nearby code, expanding scope, racking up tokens. A guide to designing stopping criteria, step budgets, and 'done' signals into agent tasks before scope creep ships as a silent failure mode.
The Confidence Score Your Users Learned to Ignore
A confidence percentage next to an LLM answer is a trust contract you usually cannot back. After a few miscalibrated 90%s, users learn to ignore the gauge — and three honest design options remain.
The Demo Worked Because You Were Watching: Session Length Is the Eval Dimension Your Suite Forgot
Five-turn demos hide the compounding error, attention drift, and sticky commitment that show up at turn twenty-eight. Treat session length as a first-class eval dimension or ship a reliability number that the user has already met the other version of.
The Embedding That Aged Out of Meaning
An embedded knowledge base silently rots as the world's vocabulary moves on. Recall dashboards miss it because they grade by yesterday's notion of similar.
The Filler Tool Call: When Agents Perform Diligence Instead of Doing Work
Production agents emit tool calls that change nothing — burning tokens, latency, and accuracy. Here is how filler emerges from training, what it actually costs, and how to engineer it out with counterfactual instrumentation and call budgets.
The First-Time User Cliff Your Aggregate Metrics Are Hiding
AI features often look healthy in aggregate while new users churn after one or two tries. Here is how the first-time user cliff hides in your metrics and the patterns that close it.
The Kill Switch Nobody Wired Because the Feature Never Failed
Launch flags get cleaned up; kill switches do not. Why every AI feature needs a durable runtime disable, a pre-decided fallback chain, and a runbook that names the lever.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
