Page 27
12 articlesThe Reroll Button as a Product Decision: When Regenerate Trains Your Users to Distrust You
The regenerate button feels like a free UX win, but ships a behavioral retrain that teaches users to treat your model as a slot machine. The design space — pagination, branching, guided regen, reroll budgets — and how to instrument reroll rate as the highest-bandwidth quality signal your product has.
The Retrieval Citation Tax: Why Compliance Adds 30% to Your RAG Token Bill
Adding citations to a RAG system looks like a one-line system-prompt change. In regulated tenants it quietly multiplies inference cost 25–40%. Here is why the tax is structural, and the architectural moves that buy most of the cost back.
The Second-Draft Agent Pattern: Why Explore-Then-Commit Beats Self-Critique
A two-pass agent shape — wasteful first draft, then clean execution from a constrained context — often beats n-of-k self-critique loops on both quality and cost.
Shadow Evals: When Private Slices Replace Your Eval Rollup
Private eval notebooks feel productive but leave the org with no rollup. The fix is a merge-gate contract: shared harness, blessed slices, named owners, and a leaderboard anyone can rerun.
Stale Few-Shot Examples and the Half-Life Your Prompt Repo Ignores
The canonical examples in your system prompt are quietly teaching the model a product that no longer exists. Eval scores stay green because the eval set decayed with them.
The Support Ticket to Eval Case Pipeline Nobody Builds
Support tickets are the highest-signal eval dataset most AI teams own, but they rot in Zendesk while the eval suite drifts in Git. Here's the four-stage pipeline that closes the loop.
Thinking Tokens Are Invisible in Your Logs and Loud on Your Bill
Reasoning tokens get billed as output but live in a field most LLM observability stacks were built before. Here is why finance finds the regressions first, and how to close the gap.
Time-of-Day Quality Drift: Why Your AI Feature Behaves Differently at 10 AM ET
Provider load is not a latency problem with a quality side effect — it is a distribution shift your eval suite never sees, and it ships a feature whose floor your team has not measured.
The Tool Schema Evolution Trap: When One Optional Parameter Changed Your Planner's Prior
A new optional parameter on an existing tool description ships clean, breaks no callers, fails no evals — and quietly inflates tool call frequency by double digits because the planner's prior shifted. Why tool schemas need semver, frequency baselines, and the same eval discipline as system prompts.
Your PRD Is an Untested Prompt — Until You Eval It
A PRD for an AI feature is a system prompt nobody compiled. Run it through an eval before sign-off and the underspecification surfaces before production does.
Agent Circuit Breakers: Why Step Budgets Are Fuses, Not Breakers
Step-count budgets are fuses that blow after the damage is done. Real agent circuit breakers combine semantic loop detection, progress signals, token-velocity ceilings, and halt-with-handoff.
Agent Memory Is a Compliance Surface: The Records-Management System You Didn't Sign Up to Build
Long-term memory in agentic products is not a feature — it is a records-management system. Provenance, deletion, audit, and residency obligations land the day the first item is written, and retrofitting them under deadline costs more than building them at design time.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
