Engineer systems that
think, act, and ship.
Practical guides on building autonomous AI systems, scaling engineering teams, and technical leadership.
Latest Dispatches
Measuring the Wrong Half of Your RAG Pipeline
A RAG system is two machines, but most eval harnesses grade only the generator. Here is why retrieval needs its own scorecard and how to build one.
Reproducing an AI Decision Eighteen Months Later
An AI decision you made eighteen months ago is nearly impossible to reproduce once the model, prompt, and retrieval index have all rotated. Here is how to capture a defensible decision record at inference time.
Retiring an Embedding Model: Re-indexing Millions of Vectors Without Taking Search Down
Swapping an embedding model looks like a config change but is a full data migration. How to re-embed a corpus, run a dual-index cutover, budget the cost and time, and prove the new index is better before your users find out it isn't.
The Agent That Deadlocked Waiting on Another Agent
Two competent agents can wait on each other forever while your bill runs. Why coordination — not model capability — breaks agent swarms, and how distributed-systems discipline fixes it.
The Compensating Transaction Your Agent Never Runs
AI agents take irreversible real-world actions with no way to unwind them. Borrow the saga pattern: pair every tool with a compensating transaction, gate the irreversible, and log before you act.
The Deprecation Notice Your Agent Can't Read
Agents don't read changelogs or Sunset headers. Here's why tool deprecation fails silently for LLM agents, and how to version tool contracts so the model actually gets the notice.
The Human Escalation Path Nobody Staffed
The escalate-to-human box in every agent diagram often points at a queue with no owner, no SLA, and no one on call. Treat escalation as a staffed product surface, not a code path.
The Feature Flag You Can't Roll Back Is a Prompt
A prompt edit changes production behavior for every user at once, with no canary and no working revert. Here's why prompts and model pins escape release discipline, and the five primitives that bring them back under it.
The FinOps Gap: Why Nobody Approved Your $40K AI Bill
Model spend enters production through a code path your change-management process never reviews. Here is why token cost is invisible to approval, and the attribution, showback, and spend-gate primitives that close the gap.
The Golden Dataset That Rots: Why Your Eval Set Drifts Out of Sync With Your Product
A golden eval set that keeps passing can be lying to you. How eval datasets rot through coverage gaps, stale labels, and distribution skew — and the hygiene that keeps the score honest.
The Idempotency Key Your Agent Forgot to Send
Agents retry side-effecting tool calls the same way they retry reads — and quietly double-charge cards and double-send emails. Here is where idempotency actually belongs, and why the prompt is the wrong place to put it.
The Latency Budget Nobody Allocated Across Your Agent's Hops
Agent SLOs are set at the boundary and left unbudgeted inside, so nobody can attribute which hop blew p99. Allocate per-hop budgets, trace at span level, and bound a tail that multiplies instead of adding.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
