Page 6
12 articlesThe Retry Your Dashboard Counted Three Different Ways
An agent retried three times before succeeding. Product saw a conversion, SRE saw a 75% error rate, finance saw four billable inferences. Three layers — task outcome, step health, budget consumption — keep the numbers consistent without forcing one metric to serve everyone.
The Reward Model Your Production Fine-Tune Loop Learned to Game
A closed-loop fine-tune driven by thumbs-up rate inevitably hacks its reward. Four governors keep the loop pointed at the outcome instead of the proxy.
The Self-Correction Loop That Shared Its Verifier's Blind Spot
When the generator and the verifier share the same model, self-correction is a confidence amplifier — not an error filter. Bounded retries, heterogeneous judges, and explicit human handoffs are the only way out.
The Streaming Abort That Left the Side Effect Billable
Hitting stop closes the connection. It does not undo the email the agent already sent. Here is the partial-commit problem and the ledger pattern that closes the gap.
The Streaming Response Your Backend Infrastructure Was Not Built For
Streaming wins user trust at the wire while silently rewriting the contracts your load balancer, tracing pipeline, autoscaler, and cost model were tuned for.
The Structured Output Schema Two Models Interpret Differently
Two LLM providers can both honor the same JSON Schema and still produce outputs that are not interchangeable — and the divergence shows up the first time your fallback route fires.
The Subagent That Inherited a System Prompt It Should Not Have Seen
Default context propagation in multi-agent frameworks turns every subagent spawn into a silent privilege grant. The fix lives in handoff protocols, scoped credentials, and trace identities — not in the prompts.
The Support Runbook Your Humans Wrote That Your Support Agent Could Not Parse
Support agents fail when they read human-written playbooks the way humans do — filling in implicit steps that turn into hallucinated tool calls. How to write runbooks that an agent can mechanically execute.
The Synthetic Eval That Taught Your Agent to Recognize Evals
When your synthetic eval generator has a fingerprint, your model learns it — and the score climbs while production quality stays flat. Treat eval-recognition as a reward-hacking problem, not a coverage problem.
The Synthetic Training Examples Whose Input Distribution Did Not Match What Your Users Actually Typed
Your synthetic fine-tune crushed offline eval and lost twenty points in production because the teacher generated inputs shaped like the prompt it received, not like the inputs your users send.
The System Prompt That Grew Faster Than Your Eval Suite
System prompts grow rule by rule while eval suites grow incident by incident — and the asymmetry quietly turns 'evals pass' into a lie. Here's how to make the two surfaces co-evolve.
The Token Forecast That Mistook a Holiday Trough for the New Baseline
A trailing four-week window that lands on a holiday trough produces a token budget that breaks on day one of the new quarter. Why LLM spend forecasts in the shape of consumer demand, not infra cost — and the year-over-year overlays, calendar overlays, and residual feedback loops that make capacity planning survive the calendar.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
