Page 143
12 articlesLong-Context Models vs. RAG: When the 1M-Token Window Is the Wrong Tool
Why dumping your entire knowledge base into a 1M-token context window fails in production — the latency, cost, and accuracy tradeoffs that make RAG the right default for most retrieval workloads, and a five-factor decision framework for when long context actually wins.
Why Your Agent Harness Should Be Stateless: Decoupling Brain from Hands in Production
Tight coupling between AI agent harnesses and sandboxes kills reliability, scalability, and security. Here's the architectural pattern that fixes it: external session logs, stateless harnesses, and isolated sandboxes.
The Model Upgrade Trap: How Foundation Model Updates Silently Break Production Systems
Foundation model updates silently break production systems through behavioral drift, changed refusal patterns, and JSON serialization inconsistencies — a practical guide to detection and safe migration.
Multi-Tenant LLM API Infrastructure: What Breaks at Scale
A production API gateway in front of LLM providers solves cost attribution and rate limit contention — but the hierarchical isolation model, token-aware limits, failover patterns, and KV cache security create complexity most teams underestimate until they're already burned.
Multimodal LLM Inputs in Production: Vision, Documents, and the Failure Modes Nobody Warns You About
A practical guide to the failure modes engineers encounter when deploying multimodal LLMs in production — from vision token cost quadratic scaling and OCR vs. native vision trade-offs to PDF table extraction, hallucination on degraded images, and composable pipeline architecture.
The Production Retrieval Stack: Why Pure Vector Search Fails and What to Do Instead
Pure vector search fails on exact keywords, rare terms, and multi-constraint queries. A practical guide to building a production retrieval stack with BM25 hybrid search, cross-encoder reranking, and stage-level metrics.
Prompt Versioning in Production: The Engineering Discipline Teams Learn the Hard Way
Managing prompt changes in production LLM systems without version control is how teams end up paged at 2am with no rollback path. A practical guide to the deployment pipeline that prevents it.
Semantic Caching for LLM Applications: What the Benchmarks Don't Tell You
Production semantic caches hit 20–45% of traffic, not the 95% vendors claim. Here's what the threshold tuning problem looks like, which failure modes practitioners miss, and when to skip semantic caching entirely.
Structured Concurrency for AI Pipelines: Why asyncio.gather() Isn't Enough
Parallel tool calls are one of the most useful LLM capabilities — but asyncio.gather() introduces orphaned tasks, silent failures, and resource leaks that only surface under production load. Here's how to do concurrency correctly in agent pipelines.
JSON Mode Won't Save You: Structured Output Failures in Production LLM Systems
Production LLM structured output fails in four distinct ways, and JSON mode only catches one of them. A breakdown of syntax, schema, semantic, and distribution failure layers — and the validation stack that handles all four.
Synthetic Training Data Quality Collapse: How Feedback Loops Destroy Your Fine-Tuned Models
Using LLMs to generate fine-tuning data creates feedback loops that amplify biases, narrow distributions, and cause irreversible model degradation — and most teams don't notice until it's too late.
The Tool Selection Problem: How Agents Choose What to Call When They Have Dozens of Tools
Why LLM agents fail at tool selection as inventories scale past a dozen tools — token explosion, retrieval failure modes, and the layered routing architecture that keeps selection accurate at 50+ tools.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
