Page 78
12 articlesTrace Sampling for Agents: Which of 10 Million Daily Spans Are Worth Keeping
Why uniform sampling defaults break for agent workloads, and the four decisions — start, end, stratification, retention — that determine your trace bill and your incident MTTR.
Accept Rate Is a Vanity Metric: Your Copilot ROI Hides in the 90 Seconds After the Keystroke
Copilot accept rates measure the frictionless half of the loop. The ROI lives in the validation tax that follows — here are the metrics that actually tell the truth.
You Accidentally Built a Feature-Flag System for Prompts — Without the Governance
Prompt config repos behave like feature-flag services at runtime but lack exposure tracking, audit logs, rollback telemetry, and per-user gating. Here is the governance gap and how to close it.
Your Accuracy Went Up and Your Calibration Collapsed
A prompt refactor that lifts accuracy three points can quietly destroy the hedging signal users trust. Measure calibration with ECE, reliability diagrams, and refusal rates — not just accuracy.
The Acknowledgment-Action Gap: Your Agent's 'Got It' Is Not a Commitment
Production agents confidently confirm actions that never happened because teams treat chat text as a contract instead of the tool call. A pattern for separating narration from commitment.
The Agent Backfill Problem: Your Model Upgrade Is a Trial of the Last 90 Days
When a smarter model disagrees with the one you shipped, every durable agent decision becomes a contested record. A framework for eval, decision, and action replay — plus the architectural prerequisites and policy matrix you need before the next upgrade.
The Agent Capability Cliff: Why Your Model Upgrade Made the Easy 95% Perfect and the Hard 5% Your Worst Quarter
Model upgrades raise your aggregate pass rate while concentrating the residual failures on the hardest 5% of traffic — here's how stratified evals and capability-frontier probing expose the cliff before it lands in your on-call rotation.
Agent Idempotency Is an Orchestration Contract, Not a Tool Property
Tool-level idempotency keys are not enough when a non-deterministic planner can re-emit the same action. The contract has to live at the orchestration boundary, keyed by structural run state — not by model-authored arguments.
Agent Latency Budgets Are Trees, Not Lines — You Have Been Debugging the Wrong Axis
Agent latency is a nested tree of planning calls, tool fan-outs, and sub-agents — flame graphs sorted by duration hide the critical path, so local optimizations miss the real budget violation. Here's how to budget, propagate deadlines, and observe slack the tree way.
Agent Memory Schema Evolution Is Protobuf on Hard Mode
Agent memory has two schemas — the store and the model's context — and only one of them migrates with a SQL script. Why protobuf's additive-only discipline is the right starting point, and what the shadow-write playbook needs on top.
Silent Success: When Your Agent Says Done and Nothing Actually Happened
Agents fail by continuing to talk. Confident prose papers over tool errors while writes never commit. The fix: demote the model's claim to a hypothesis, promote tool responses and post-action probes to authoritative signals, and measure effect landing instead of turn success.
The Agent Paged Me at 3 AM: Blast-Radius Policy for Tools That Reach Humans
Granting an agent PagerDuty access is an infra decision with product-team consequences. A control plane for human-facing tools — rate limits, dry-run, off-ramps — that prompts can't enforce.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
