Page 71
12 articlesWhen Your Evals Disagree: A Signal Hierarchy for the Week the Numbers Contradict Each Other
Four eval signals on a prompt change rarely agree. Without a written hierarchy for which signal wins under which conditions, every release week becomes a debate about whose number to trust.
Few-Shot Rot: Why Yesterday's Examples Hurt Today's Model
Few-shot examples are tuned to a specific model. After a model upgrade, the demonstrations that lifted accuracy can quietly start dragging it down — here is the audit and provenance discipline that prevents rot.
Found Capabilities: When Users Ship Features Your Team Never Roadmapped
Every sufficiently capable model exposes behaviors your team never roadmapped. Users find them, build workflows on top, and treat the next model upgrade as a regression. Here's the product discipline that turns found capabilities into decisions you actually own.
Generative UI as a Production Discipline: When the Model Renders the Screen
When the model emits component trees instead of text, design review, accessibility audits, and prompt-injection threat models all have to be rebuilt from scratch.
The Ghost Employee in Your Audit Log: Agents With Borrowed Credentials Break IAM
Borrowed credentials make agents look like the humans who launched them in every audit log — and that thin film is why a prompt injection in 2026 becomes an unattributable breach.
GPU Capacity Planning When Demand Is a Cliff, Not a Curve
Agent workloads break smooth-curve capacity planning. Plan in tokens, treat fanout as a first-class metric, and reserve for cliffs you know will land.
The Hollow Explanation Problem: When Your Model's Reasoning Is Decoration, Not Evidence
Generated explanations next to LLM outputs often have no causal link to the actual computation. Why post-hoc rationalization erodes user trust faster than admitted uncertainty, and the design patterns that don't fake explainability.
Inter-Token Jitter: The Streaming UX Failure Your p95 Dashboards Can't See
Time-to-first-token and total completion time both pass SLO while users complain the AI 'froze' mid-response. The metric your dashboard hides is the gap between consecutive tokens — and smoothing it is a UX problem, not a throughput problem.
The Internal LLM Gateway Is the New Service Mesh
At fifty engineers, every team rebuilds the same LLM gateway badly. Why the pattern keeps emerging, what to centralize vs leave at the edge, and how the political fight gets settled.
The Inverted Agent: When the User Is the Planner and the Model Is the Step-Executor
Most agent products put the model in charge of planning and the user in charge of approving. For high-stakes work, that polarity is exactly backwards — and the fix is a different product, not a better prompt.
JSON Mode Is a Dialect, Not a Standard: The Silent Breakage in Your Fallback Path
Every major LLM provider ships JSON mode under the same name and a different contract. The day your fallback router activates is the day you find out which differences your parser couldn't survive.
The Eval Pickle: When Your LLM Judge Gets Smarter Than the Model It Grades
When the LLM grading your evals gets sharper, your scores drop on a system that didn't change. Here's how to tell judge drift from model regression — and stop debugging the wrong instrument.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
