Page 84
12 articlesRetry Amplification: How a 2% Tool Error Rate Becomes a 20% Agent Failure
Agent loops turn a 2% tool error rate into a 20% user-visible failure by multiplying retries across steps and SDK layers. Here is the math, the self-DoS pattern, and the retry budget discipline that stops it.
The Right-Edge Accuracy Drop: Why the Last 20% of Your Context Window Is a Trap
Filling an LLM's advertised context window wrecks accuracy at the right edge — the failure mode past 'lost in the middle,' with benchmarks, safety margins by task, and prompt fixes.
The Rubber-Stamp Collapse: Why AI-Authored PRs Are Hollowing Out Code Review
When most diffs in a repo start life as model output, reviewers anchor on 'looks plausible' and miss the semantic bugs that don't render as syntactic smell. The countermeasures, the disclosure question leadership has to answer, and the incident curve that catches up six months later.
Sampling Bias in Agent Traces: Why Your Debug Dataset Silently Excludes the Failures You Care About
Head-based and uniform-random sampling silently excise the rare catastrophic agent trajectories from your debug corpus. Tail sampling, anomaly-keyed retention, and per-failure-mode reservoirs build a debug dataset that actually contains the failures you need.
Semantic Cache Is a Safety Problem, Not a Perf Win
Semantic caches can serve another user's response in under a millisecond and your hit-rate dashboard will turn green doing it. The cache-key design, provenance envelope, and audit trail that prevent cross-user leak by construction.
Semantic Diff for Prompts: Why Git Diff Lies About What Your Prompt Change Will Do
Text-level diffs have almost no correlation with how an LLM's behavior changes. A three-word edit can flip 30% of outputs while a fifty-line restructure changes nothing. Here is how to build a semantic diff toolkit that PR reviewers can actually trust.
The Ship-and-Pin Trap: How Model Version Stability Becomes Deprecation Debt
Pinning a model version buys short-term stability and quietly accrues deprecation debt. Scheduled re-qualification, drift monitoring against the next tier, and a dual-track prompt portfolio turn migrations into routine operations instead of fire drills.
Spec-First Agents: Why the Contract Has to Land Before the Prompt
Prompt-as-spec collapses under more than one author. A spec-first contract — inputs, outputs, invariants, errors, refusals, escalations — turns prompt edits into diffs, makes evals derivable, and shrinks owner onboarding from months to a week.
The Synthetic Preference Trap: How AI-Ranked RLHF Quietly Drifts Your Model Into the Teacher's Voice
Synthetic preference data feels like a free lunch — until your product quietly starts sounding exactly like the teacher model you trained it from. A field guide to spotting, measuring, and bounding RLHF flavor drift.
Token Spend Is a Security Signal Your SOC Isn't Watching
Anomalous LLM token spend is the earliest signal of a compromised API key, prompt injection, or data exfiltration — but billing owns the dashboard and security owns the response. Here is how to wire them together.
Your Tool Descriptions Are Prompts, Not API Docs
Tool spec text is the prompt the model reads before deciding when to invoke. Treat it like a prompt — concrete use cases, negative examples, sibling disambiguation — not like OpenAPI docs.
Tool Hallucination Rate: The Probe Suite Your Agent Team Isn't Running
Most agent teams measure tool-call success but never measure tool hallucination. Split the rate into three — unknown-tool, shadow-call, hallucinated-argument — and build the probe suite that catches each before production does.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
