Page 19
12 articlesContext Length Is a Security Boundary, Not Just a Cost Line
A long enough conversation buries your system prompt under fresher tokens until guardrails quietly fail. Why context length belongs in the threat model — and how to control it.
The Context Window Is a Commons, and Every Team Is Grazing It
An agent's context window is a shared, depletable resource with no allocator. Here is why per-feature additions are locally rational and globally ruinous, and how to govern it with attribution, quotas, and audits.
The Degradation Signals Your Agent Never Receives
An agent calling a downstream API sees only the response to its last request — no status page, no changelog, no warning banner. Here is why agents run straight into brownouts and rate limits, and how to build the side-channel that carries the operational signals they were never given a way to hear.
The Demo That Set a Baseline You Cannot Afford to Run
An agent demo runs on the frontier model, hand-picked inputs, and no load — then quietly becomes the baseline leadership expects. Here is how to price the demo-to-production gap before it becomes a promise.
The Demo-to-Production Cliff: Why a 90%-Accurate Agent Ships at 0%
An agent that succeeds 90% of the time per step is a great demo and an unshippable product. The gap is not a polish problem — it is a tail of expensive failures, and the fix is making that tail cheap.
Dogfooding Is Not an Eval Strategy
Your team using the product every day is a smoke test, not an eval. Why builders are the worst sample of their own users, and how to measure AI product quality on the traffic that actually breaks it.
The Embedding Upgrade That Silently Re-Ranks Your Entire Corpus
Swapping your embedding model for a higher-benchmarking one invalidates every vector you have stored. Why the upgrade silently degrades retrieval, and how to migrate it like a schema change.
The Eval Set That Got Easier While You Weren't Looking
An LLM eval suite that everything passes has stopped measuring anything. Why static eval sets saturate, how to spot it, and how to keep a usable score gradient.
The Eval Set Is a Lagging Indicator: Your Green Dashboard Only Knows Last Quarter's Failures
An eval suite grown only from postmortems certifies your AI system against the past. Here is why a green pass rate lies on migration day, and how to fund exploratory coverage.
Your Eval Set Is a Frozen Photograph of Traffic Your Users Already Left
A benchmark gain measures progress on a distribution your users already left. How eval-set staleness, the survivorship trap, and a single aggregate score hide a silent decline — and how to keep your eval tracking the river.
The Eval Suite That Became the Spec Nobody Agreed To
Most agent teams have no requirements doc — the eval suite became the spec by default. Why a green eval run certifies one engineer's assumptions, and how to give the eval set the review rigor of an API schema.
The Fallback Model You Never Load-Tested
A configured fallback proves your router works — it proves nothing about whether your application survives the secondary model's output. Why named fallbacks fail under real traffic, and how to test the failover before your provider does it for you.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
