Page 23
12 articlesWarm Pools and Cold Truths: The Hidden Latency Floor of Serverless LLM Inference
Scaling GPU inference to zero converts a steady dollar cost into a spiky latency cost hidden in the p99 tail. Here is the break-even math and the mitigation toolkit.
When No One Answers the Escalation: Human-in-the-Loop Is a Staffing Problem
Human-in-the-loop assumes a person answers the escalation. In production it is a queue with arrival rate, service time, and abandonment — and an unanswered escalation is worse than none.
When the Cheap Model Is the Expensive One
Routing traffic to a smaller model lowers cost per token but can raise cost per finished task. Here is where the savings leak back out — and how to measure it before you ship.
Who Gets Paged When the Agent Is Wrong: On-Call for Non-Deterministic Systems
Agent failures don't reproduce, don't roll back, and stay green on every infrastructure dashboard. Here is how to rewrite the runbook, the alerts, and on-call expectations for systems you can't single-step.
Who Owns the Idle Cost of an AI Feature
Provisioned throughput, reserved GPUs, and warm vector indexes bill whether or not traffic arrives. Idle cost grows because it falls in the org seam between product, infra, and finance — here is how to make the gap visible and owned.
Who Pays for the Model's Mistake: Designing Liability Into Agent Products
Agent errors are not support escalations — they are billing events with a real owner. How to design a liability model, provenance trails, reversibility tiers, and mistake-cost evals before the first angry ticket.
You Can't Email a Changelog to a Model: Why API Deprecation Breaks When the Caller Is an LLM
An LLM caller has no inbox, no stable identity, and no obligation to read your migration guide. Here is why the fifteen-year-old API deprecation playbook fails for agents — and what to change in the tool schema, error messages, and gateway instead.
The 14-Month Half-Life of Your Prompt Expert
The engineer who holds your prompt and eval knowledge is being repriced by the market faster than your comp band moves. Here is why the generic IC ladder cannot see them, and what to change before they leave.
Agent Memory Is a Cache With No Invalidation Policy
Long-term memory ships as a feature, but it is a cache of facts about a world that keeps changing. Without invalidation, provenance, and conflict rules, it becomes a slow-motion correctness bug.
The Confidence-Score Tax: Why Asking the Model How Sure It Is Costs More Than Being Wrong
Adding a confidence field to an LLM output feels free. It is not. Here is the per-request tax it imposes, why the number is rarely calibrated, and what to measure before routing production traffic on it.
Deleting an Eval Case Is a Decision, Not Cleanup
Pruning an eval suite for speed or cost looks like maintenance, but every deleted case retires a guarantee the team can no longer see. Borrow the API deprecation lifecycle to retire eval cases deliberately.
The PM-Eval Translation Gap: When Ship Decisions Outrun the Vocabulary
An eval score is a lossy compression of AI quality, and the PM who owns the launch often can't decompress it. Here is the literacy bridge that keeps ship decisions anchored to the data instead of the loudest voice.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
