Page 13
12 articlesThe Pager Doesn't Ring for a 3% Regression: On-Call for Statistical Failures
Traditional on-call catches crashes and latency spikes. The failures that actually hurt AI products are quiet 3% quality drops that no threshold ever trips. Here is how to build alerting for statistical failures.
GPU Scheduling Is a Queueing Problem, Not a Provisioning Problem
Slow LLM endpoints are usually a queueing failure, not a hardware shortage. How continuous batching, KV-cache limits, and prefill/decode scheduling set your tail latency — and why adding GPUs won't fix it.
Token Budgets Are a Headcount Decision in Disguise
Every choice about how many tokens a prompt spends is a hidden trade against engineer time, support load, and infra. A framework for making that trade explicit instead of letting it accrete.
Your AI Roadmap Needs a Decommission Column
Launch-only roadmaps quietly accrue liability in AI systems, where models, prompts, and eval sets rot on clocks you don't control. Add a decommission column and treat sunset as a first-class deliverable.
Your Provider's Safety Filter Is Product Policy You Can't Edit
Your LLM provider's safety filter is product policy written by someone who never met your users. How over-refusal, quiet policy updates, and uniform moderation erode AI products in specialized domains — and how to engineer the seam back under your control.
The Build-vs-Buy Line Moved: Deciding AI Features When Provider Primitives Absorb Your Infra
Provider APIs keep absorbing the retrieval, memory, and structured-output layers you used to build. A four-quadrant framework for deciding what to build, what to rent, and where your actual moat lives.
The Demo-to-Production Tax: The 90% of AI Work the Prototype Hid
The AI demo that wowed the room was 10% of the work. The other 90% — evals, guardrails, observability, cost controls, fallback paths — is the tax that kills 88% of pilots before production.
Renting Intelligence: A CFO's Mental Model for LLM Spend
Token spend behaves like cost of goods sold, not R&D — and pricing a usage-scaling cost as if it were fixed is what blindsides finance teams. A practical model for forecasting LLM spend, spotting the margin trap, and deciding when renting intelligence pays for itself.
The Meeting That Should Have Been an Eval
The recurring 'is the model good enough?' debate never converges because the team argues from anecdotes. Here is how to convert that subjective shipping argument into a standing offline metric the whole team is bound by.
The Seniority Inversion: Why Your Senior Engineers Slowed Down When the Agents Sped Up
Agents made code generation cheap and review expensive. The cost landed on the one resource that doesn't scale: senior judgment. Why the load concentrated, and how to rebalance it.
Capacity Planning for Agents: Why Concurrency, Not QPS, Is Your Real Unit
QPS hides agent load because runs stay in flight for minutes and fan out. Use Little's Law to size for concurrency — the number actually holding your resources.
How PII Redaction Sentinels Quietly Collapse Your Vector Index
Privacy redactors that replace PII with sentinel tokens can silently dominate your embedding geometry, collapsing every redacted document into a single hub of the vector index and degrading retrieval where no benchmark is watching.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
