Engineer systems that
think, act, and ship.
Practical guides on building autonomous AI systems, scaling engineering teams, and technical leadership.
Latest Dispatches
How PII Redaction Sentinels Quietly Collapse Your Vector Index
Privacy redactors that replace PII with sentinel tokens can silently dominate your embedding geometry, collapsing every redacted document into a single hub of the vector index and degrading retrieval where no benchmark is watching.
The MCP Tool List Grew Mid-Session and Your Agent Called a Tool It Had Never Been Told About
An MCP server can grow its tool list between two calls in the same session, and the agent's next selection can land on a tool the client was never told about — a hallucination that resolves to a real action.
The A/B Test Winner Whose Verbose Output Triggered Your Click Handler More Than the Better Answer
Why prompt experiments on engagement metrics tend to ship the longer variant, and the patterns that decouple response shape from response quality before a satisfaction regression forces a reckoning.
The Agent Memory Store That Survived Your Tenant Deletion Because Nobody Owned It
A correct deletion saga is no defense when the inventory it iterates over is stale. Why your tenant-deletion guarantee silently decays with every release that ships a new persistent store nobody filed.
The Agent Timeout Your Users Learned to Game for Refunds
A generous timeout-refund policy on an agent platform created a behavioral cohort that systematically gamed the boundary, doubled the timeout rate, and disguised itself as a quality regression.
The Agent Wall-Clock Budget That Raced Your Tool's Own Timeout
Inside the agent stack, the agent's clock and the tool's clock almost never share a t-zero. When their budgets drift, an 8-second tool call lands against a 7.9-second deadline and the harness replans around a success it never saw.
The Async Tool Call That Resolved After the User Already Closed the Conversation
Long-running tool calls outlive the chat sessions that dispatched them. When the user closes the tab, the result still arrives — to a conversation that no longer exists, to the wrong session, or to nobody at all.
The Canary Cohort Your Rollout Hashed by ID That Clustered Power Users Into One Arm
A uniform hash of opaque IDs is not a uniform sample of users. When ID assignment correlates with engagement, a hash-bucketed canary can quietly assign every power user to one arm and report a phantom win.
The Citation Index Your Chunker Shifted by One When It Started Prefixing Line Numbers
A document chunker added a [line N] prefix and every citation started pointing one paragraph before the evidence — the failure mode where two systems agree on the shape of an integer but disagree on its meaning, and how to catch it before an auditor does.
The Citation URL That Resolved But No Longer Said What the Model Quoted
A RAG citation passes link-checks and still loses an audit, because reachability is not fidelity. How to snapshot, hash, and retain cited spans so a transcript survives the source's editor.
The Codebase Index Your Coding Agent Rebuilt From a Checkout Three Weeks Behind Main
When a coding agent's semantic index and its working tree drift apart, the agent grounds confident claims on code that no longer exists — and the failure mode hides in ordinary-looking PRs.
The conversation_id Collision That Swapped Two Users' Contexts at the Gateway
A flat conversation_id namespace plus drifting per-tier UUID generators can swap two users' contexts at the gateway. Treat conversation IDs with the rigor your payments team applies to transaction IDs.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
