Page 22
12 articlesThe Eval That Quietly Went Stale: When Your Test Suite Measures a World That No Longer Exists
A green eval run can certify the past instead of the present. How eval suites decay, how to tell a real regression from an outdated test, and how to build freshness into the suite itself.
Token Budgets Are a Scheduling Problem, Not a Prompt Problem
Agent quality decays when one verbose tool result starves the context window. Treat token budgets like OS memory: set ceilings, evict by priority, and reserve room to reason.
The Tool Default Argument Is a Policy Decision in Disguise
Every optional argument your agent skips inherits a default you chose. Those defaults are unaudited policy — invisible in the trace and unowned in review.
Your Tool Descriptions Are an Instruction Channel the Model Obeys
Tool descriptions are prose the model treats as authoritative instructions, yet code review and input sanitization never inspect them. Here is how poisoned metadata and rug-pull attacks slip through, and the discipline that closes the gap.
The Tool Schema You Changed Without Telling the Agent
A renamed field is a routine API change for your backend — and a silent breaking change for the LLM that calls the tool. How to treat a tool schema as a versioned contract with two consumers.
The Tool That Worked Until Two Agents Called It At Once
Agent tools pass single-caller tests and break the day a second agent shows up. Why concurrency bugs are structurally invisible to serial evals, and how idempotency, locking, and load tests fix them.
The Distributed Trace That Goes Dark at the Agent Handoff
Your span tree is clean right up to the moment one agent calls another — then it goes dark exactly where the bug lives. Here is why agent handoffs break trace context, and how to make the handoff carry it.
When Two Agents Share a Tool: Concurrency Bugs in Multi-Agent Systems
Spinning up a second agent turns you into a distributed systems engineer. Race conditions, lost updates, and dirty reads return as silent corruption — and how to design the tool layer to stop them.
Halted Is Not a Status: Why Agents Need a Typed Terminal-Reason Protocol
Agent dashboards report completion rate, but a halted run that gave up looks identical to one that succeeded. A typed terminal-reason protocol makes how an agent ends a first-class, monitorable signal.
The Undo Button Your Agent Assumes Exists
AI agents plan as if every action can be undone, a habit learned in reversible code sandboxes. Encode reversibility tiers into your tools so one-way doors stay safe.
Your Vector Index Is a Cache With No Invalidation Strategy
A vector index is a derived copy of your source data, which makes it a cache that goes stale: edits never propagate, deleted documents leave ghosts, and revoked permissions leak. Why RAG reliability is a cache-invalidation problem, not a similarity-search one.
The Vector Index Has a Staleness SLO Nobody Set
A nightly re-indexing job is a freshness promise nobody wrote down. How to turn vector index lag into a measured SLO, surface data age to agents and users, and re-index by decay rate instead of by habit.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
