Page 144
12 articlesVoice AI in Production: Engineering the 300ms Latency Budget
Why voice AI feels robotic even when the model sounds good — and the streaming pipeline architecture, turn detection strategy, and transport choices that get you under 300ms.
When Thinking Models Actually Help: A Production Decision Framework for Inference-Time Compute
A decision framework for when reasoning models like o1, o3, and Claude extended thinking actually improve production outcomes — and when they burn tokens without improving results.
The Three Memory Systems Every Production AI Agent Needs
A practical guide to episodic, semantic, and procedural memory in AI agents — and why treating all persistent state as a single vector store will eventually break your production system.
What Nobody Tells You About Running MCP in Production
A practical guide to MCP's hidden production challenges — transport selection, tool schema design, tool poisoning attacks, and the gateway pattern that actually scales.
Beyond JSON Mode: Getting Reliable Structured Outputs from LLMs in Production
Prompt-only JSON extraction fails 5–20% of the time in production. A practical breakdown of all four generations of structured output techniques — from JSON mode to constrained decoding — with library recommendations and schema design rules.
What Your APM Dashboard Won't Tell You: LLM Observability in Production
Production LLM systems fail silently — green dashboards hide hallucinations, prompt drift, and wrong tool selection. Here's the instrumentation model that actually surfaces what's going wrong.
LLM Routing and Model Cascades: How to Cut AI Costs Without Sacrificing Quality
Running every query through a frontier model is the most common way teams overspend on AI. LLM routing and model cascades can cut costs by 45–85% while maintaining 95% of quality — here's how the patterns actually work in production.
Writing Tools for Agents: The ACI Is as Important as the API
How you design tools for AI agents — schemas, descriptions, return values, error messages — directly determines agent reliability. A guide to treating the agent-computer interface as seriously as any production API.
Fine-Tuning Is Usually the Wrong Move: A Decision Framework for LLM Customization
Most teams reach for fine-tuning too early. Here's a practical decision framework — backed by benchmarks and production examples — for when prompt engineering beats fine-tuning, when it doesn't, and the real economics of each approach.
Streaming AI Applications in Production: What Nobody Warns You About
Production streaming failures almost never come from the LLM itself — they come from NGINX buffering silently, load balancers timing out long-lived connections, and incremental JSON parsers degrading to O(n²). A practical guide to the infrastructure patterns that actually break at scale.
In Defense of AI Evals, for Everyone
Most teams that claim to skip evals are already doing evaluation — just badly. Here's why systematic AI evaluation matters, when lighter approaches are defensible, and how to run evals that surface real signal.
Data Flywheels for LLM Applications: Closing the Loop Between Production and Improvement
A production data flywheel turns user interactions into model improvements — but less than 1% of interactions yield explicit signal, and naively training on that 1% quietly poisons your system. Here's the architecture, feedback signals, and failure modes that determine whether your loop compounds or collapses.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
