Page 29
12 articlesThe Streaming Token the User Acted On Too Soon
Token-by-token streaming makes assistants feel fast, but it also exposes the model's unfinished thinking as a finished answer. Here is the race condition that causes it and the design patterns that fix it.
Structured Output Is Not Validated Output
JSON mode and constrained decoding guarantee the shape of an LLM response, not its meaning. Why a passing schema check is the start of correctness work, and where semantic validation actually belongs.
Your System Prompt Grows After Every Incident — and Nobody Deletes a Line
Every production incident leaves a defensive sentence in your system prompt, and nobody ever deletes one. Here is why prompt accretion is real technical debt and how to prune it with dating, half-lives, and ablation.
Task Completion Goes Green While Users Quietly Suffer
A 94% task-completion dashboard can stay green while an agent burns tokens, backtracks, and exhausts users. Why completion is the wrong number and four trajectory metrics that see what it cannot.
The Test the Agent Wrote That Tests Nothing
Coding agents generate test suites that pass, raise coverage, and catch nothing. Why agent-written tests drift into tautologies, and how mutation testing and red-green discipline make them constrain behavior again.
When Your Test Set Leaks Into Fine-Tuning: The Contamination You Cause Yourself
Benchmark contamination is usually blamed on model vendors, but the worst leaks are the ones your own team creates — failure triage, synthetic data, and shared RAG corpora that quietly move eval cases into training.
The Agent That Remembers What You Took Back: Deletion as a First-Class Memory Operation
An append-only agent memory store rots the moment a stored fact becomes false. Why deletion, retraction, and invalidation must be first-class operations — and how to design memory writes that can be found, contradicted, and removed.
The Eval That Quietly Went Stale: When Your Test Suite Measures a World That No Longer Exists
A green eval run can certify the past instead of the present. How eval suites decay, how to tell a real regression from an outdated test, and how to build freshness into the suite itself.
Token Budgets Are a Scheduling Problem, Not a Prompt Problem
Agent quality decays when one verbose tool result starves the context window. Treat token budgets like OS memory: set ceilings, evict by priority, and reserve room to reason.
The Tool Default Argument Is a Policy Decision in Disguise
Every optional argument your agent skips inherits a default you chose. Those defaults are unaudited policy — invisible in the trace and unowned in review.
Your Tool Descriptions Are an Instruction Channel the Model Obeys
Tool descriptions are prose the model treats as authoritative instructions, yet code review and input sanitization never inspect them. Here is how poisoned metadata and rug-pull attacks slip through, and the discipline that closes the gap.
The Tool Schema You Changed Without Telling the Agent
A renamed field is a routine API change for your backend — and a silent breaking change for the LLM that calls the tool. How to treat a tool schema as a versioned contract with two consumers.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
