Page 74
12 articlesThe Router Is the Product: Why Your Cheap Classifier Decides More Behavior Than Your Flagship Model
Cost-aware LLM routing makes the cheap model the actual product surface for most users. If your eval discipline still points at the flagship, you are flying blind on 70% of traffic — here is the router-as-product framing that fixes it.
Sampling Parameter Inheritance: When Temperature 0.7 Leaks From the Planner Into the Verifier
Agent harnesses that propagate temperature down the call tree turn the planner's creativity knob into the verifier's bug. Per-role sampling profiles, default-deny inheritance, and the disagreement-rate eval that catches the leak.
Session Stitching: Why Your Conversation-ID Is a Lie
Frameworks ship session-ids; users live in tasks. The gap between them is where half of agent UX disappears, and the fix is a task-id, not longer sessions.
Your Shadow Eval Set Is a Compliance Time-Bomb
Production-trace eval pipelines accumulate PII no one promised users would be processed this way. The fix is sanitization at the write boundary, schema-typed spans, and tag-based retention — not regex scrubbers at read time.
Shadow MCP: The Tool Servers Your Security Team Has Never Heard Of Are Already Running on Your Engineers' Laptops
MCP made it trivially cheap to wire a developer laptop into prod-adjacent systems. The artifact is a loopback socket using credentials the engineer already has — invisible to procurement, CASB, and SSO logs. The discovery and governance discipline that has to land before the first breach disclosure.
The Shared-Prompt Flag Day: When One Edit Becomes Thirty Teams' Regression
Centralizing a safety preamble looks like a clean DRY win until the first edit ships and thirty consumer teams' evals tank. Here's why shared prompts behave like distributed systems, and the governance scaffolding that survives the first flag day.
Speculative Decoding Is a Streaming Protocol Decision, Not an Inference Optimization
Speculative decoding promises identical model output at 3-6x speedup, but that guarantee binds tokens leaving the inference engine — not bytes already shown to the user. When you stream draft tokens before verification, rejected suffixes have to be retracted, and which surfaces tolerate retraction is a product decision the inference team rarely scopes.
The AI Feature Metric Trap: Why DAU and Retention Lie About Stochastic Surfaces
DAU, conversion, and retention were built for click streams. AI features emit task arcs — request, response, follow-up, resolution — and the dashboard you imported from the deterministic playbook will tell you the feature is winning while users route around it.
Your stop_reason Is Lying: Building the Real Stop Taxonomy Production Triage Needs
Vendor stop_reason values give you four buckets when production triage needs eight. Here is how to build the parallel stop-taxonomy that turns a black-box termination into a debuggable signal.
Streaming JSON Parsers: The Gap Between Tokens and Typed Objects
JSON.parse is all-or-nothing, but LLM token streams are not. Why streaming structured output is one design problem the API and the SDK have to solve together — and what a real partial parser must do.
Structured Concurrency for Parallel Tool Fanout: Who Owns Partial Failure?
Most agent frameworks run parallel tool calls as detached goroutines, then rediscover the failure modes structured concurrency solved two decades ago — partial failure, honored cancellation, runaway cost.
Synthetic Users for Multi-Turn Agent Eval: When Your Test Fixture Has To Push Back
Single-turn evals miss the multi-turn failure modes that matter. LLM-driven user simulators with personas, patience budgets, and abandonment thresholds run thousands of conversations a night — but only when the simulator-vs-production gap is calibrated, not assumed.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
