Page 30
12 articlesThe Eval-Prod Gap: Detecting Behavioral Mode Switching in Production LLMs
Production LLMs routinely behave differently in evaluation contexts than in real traffic — and most teams never detect it. Here's how to surface the divergence before it erodes trust in your system.
The Expertise Cliff: Why AI Coding Agents Fail in Mature Codebases
AI coding agents produce genuine velocity gains on greenfield code—then quietly accumulate damage in mature systems. The gap is tacit knowledge: the undocumented constraints, rejected alternatives, and architectural rationale that live in engineers' heads but never reach the repository.
The Feedback Provenance Gap: Why Your Training Signal Might Not Be What You Collected
When feedback enters your AI improvement loop, it passes through filtering, weighting, and upsampling steps with no audit trail. How to build the provenance infrastructure that makes training signal corruption traceable before it silently degrades your model.
The First AI Feature Problem: Why What You Ship First Determines What Users Accept Next
User trust in AI is formed on the first failure, not the first success. The sequence of your AI feature launches matters more than the quality of any individual feature — and getting it wrong is harder to recover from than most teams expect.
Graph Reasoning Gaps in LLMs: Scaffolding Relational Tasks That Fool Sequence-Trained Models
LLMs trained on sequences systematically fail on graph-structured reasoning tasks. Here's the engineering pattern that compensates: structured encoding, tool-based traversal, and a pre-build diagnostic to detect whether you're fighting the architecture before you write your first prompt.
The Inference Fleet: Applying SRE Discipline to Multi-Provider LLM Dependencies
Production AI stacks now span multiple providers, fine-tuned endpoints, and self-hosted models. Managing them requires SRE-style fleet discipline: service catalogs, per-provider SLO tracking, capacity planning, and clear ownership models.
Intent Drift in Long Conversations: Why Your Agent's Goal Representation Goes Stale
User intent shifts across long sessions in ways that accumulated context flattens into a single static goal. Here's how agents lock onto early signals, misread corrections as clarifications, and what to do about it.
The Invisible Handoff: Why Production AI Failures Cluster at Component Boundaries
Most production AI failures don't happen inside the model—they happen at the invisible seams where one component's output becomes another's input. Here's how to find and fortify those boundaries.
Knowledge Age Routing: Matching Queries to the Right Temporal Layer in Production AI
Production AI systems carry knowledge at four freshness levels—parametric weights, RAG indexes, session context, and live retrieval. Routing queries to the wrong layer produces confident wrong answers with no visible error signal.
The Confident Hallucinator: Runtime Patterns for Knowledge Boundary Signaling in LLMs
LLMs confidently hallucinate because RLHF trains them to sound certain. Here's how to detect knowledge boundaries, route by confidence, and build fallback chains that make uncertainty actionable in production.
Why Your AI Sounds Wrong Even When It's Technically Correct
Technical correctness and communicative appropriateness are orthogonal failure modes. Register mismatch is a silent churn driver that hides behind vague user feedback — and almost never shows up in your eval suite.
The LLM-as-Compiler Pattern: Separating Plan Generation from Execution
Prompting an LLM to emit a structured execution plan that a deterministic engine runs — instead of letting it act step-by-step — delivers 50% higher accuracy at one-eighth the cost. Here's when the pattern is worth the overhead and how to implement it in production.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
