Page 64
12 articlesConfidence Strings, Not Scores: Why Your 0.87 Badge Moves Nobody
A 0.87 confidence badge changes no user behavior. A natural-language hedge that names what the model didn't check changes a lot. Why probability scores are the wrong shape of signal, and how to ship uncertainty as content instead of a UI overlay.
Cost-Per-Correctness, Not Cost-Per-Token: The Unit Metric Your Bill Won't Tell You
Token spend is the numerator. Eval-graded outcomes are the denominator. Tracking only the bill is how cheap-tier migrations silently regress quality and inflate downstream support cost.
Cross-Team Agent SLAs Don't Compose: The 99% Math Your Org Forgot to Budget
When agents call agents across team boundaries, individual SLOs stop predicting end-to-end behavior. The four pieces that have to land before the composition math eats your reliability budget.
The Eval Bottleneck: Your Eval Engineer Is Now the Roadmap
In 2026 the throughput limit on AI features isn't model shipping or prompt iteration — it's eval engineering. Here's the staffing ratio, platform investment, and leadership reframing required before your only eval engineer quits.
Eval Differential as Branch Protection: Ship Score Diffs, Not Score Floors
Score floors let silent regressions ship while flagging real improvements. A baseline-aware, slice-level eval diff turns the eval gate into a regression detector your team can trust.
The Eval-Set Poison Pill: When Your Benchmark Becomes a Backdoor
Most teams trust the eval because nobody owns auditing it. The labeling pipeline is a human supply chain — and the gold set inherits whatever distortion the humans introduce.
Eval Sets Have Seasons: Why Quality Drops on the First Monday of Tax Season
Production traffic is not stationary. An eval set sampled in March, run in October, scores an October-shaped customer who never appeared in the gold rows. Here is how to keep the gate honest.
Your Gold Eval Set Has Drifted and Its Pass Rate Is the Reason You Can't See It
Gold eval pass rates stay green while production drifts away from them. Run a shadow eval built from current traffic in parallel — the disagreement metric is the drift detector your dashboard is missing.
The Human Attention Budget Is the Constraint Your HITL System Silently Overspends
HITL systems treat reviewer time as infinite, but vigilance decrement and automation bias quietly turn the safety net into a rubber stamp. Design for the real human limits.
The Idle Agent Tax: What Your AI Session Costs While the User Is in a Meeting
Long-lived AI agent sessions keep accruing cost even when the user is in a meeting. Here is what those idle minutes actually pay for, and how to design hibernation tiers that hold latency without burning the bill.
Your Inference Chargeback Is Quietly Taxing Eval Discipline
Pricing inference tokens but not eval coverage rewards model upgrades and punishes evaluation, so eval coverage shrinks while the bill grows — exactly opposite to FinOps intent.
Inference Cost Forecasting: The Capacity Plan Your Finance Team Wants and You Can't Write
Classical capacity planning assumes a measurable workload and a stable unit cost. AI workloads break both — and the SaaS-style forecast you hand finance is the reason they keep calling for a re-baseline. Here's the four-discipline FinOps shape it should take instead.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
