Page 39
12 articlesThe AI Incident Postmortem Nobody Writes: A Four-Layer Diagnosis Framework
When an AI feature causes a production incident, standard postmortems fail. Here's a four-layer diagnosis framework — model, data, integration, infrastructure — that lets teams assign accountability without blame diffusion.
AI Output Volatility Is a Business Risk You're Probably Underpricing
Building pricing tiers, SLAs, and customer commitments on top of a probabilistic system is carrying undisclosed risk. Here's how to quantify it and hedge against it.
Your System Prompts Are Still in English: The Silent Cost of Incomplete AI Localization
Translating UI strings while keeping system prompts in English silently degrades non-English users. How the failure compounds across formality, structured outputs, tokenization, and invisible eval gaps — and what to do about it.
Your AI Feature's Quiet Quitters: How to Detect Silent User Distrust
Most AI feature failures are invisible in aggregate metrics. Users don't file tickets or disable features — they quietly route around them. Here's how to detect the behavioral signals that reveal silent trust abandonment before it shows up in your retention curve.
Training Your AI on Production Data Without Triggering a Legal Blocker
How behavioral telemetry for AI model improvement collides with GDPR and CCPA — and the federated learning, differential privacy, and consent architecture patterns that let you keep the feedback loop without triggering a legal blocker.
API Documentation Is Reliability Infrastructure: How Your Docs Determine Agent Success Rates
When AI agents consume your API via tool calling, documentation quality becomes a direct reliability variable. Ambiguous parameters and missing error semantics cause measurable failure rates that no amount of prompt tuning can fix.
Code-Specific RAG: Why General Retrieval Fails for Codebases
Token-based chunking destroys code's structural properties before the retriever ever sees them. AST-aware chunking, call-graph traversal, and test file co-location are the patterns that actually work for codebase retrieval.
The Context Format Decision Most Teams Make Accidentally: JSON vs Markdown vs Plain Text
Choosing between JSON, markdown, and plain text for LLM context isn't a stylistic preference — it determines reasoning mode, accuracy, and cost. Here's how to make the decision deliberately.
The AI Code Feedback Loop: How Today's Generated Code Trains Tomorrow's Models
As AI-generated code floods production codebases, it becomes training data for the next model generation. The feedback loop is already measurable — and the failure mode is subtle enough to arrive undetected.
Why AI Features Break A/B Testing (and the Causal Inference Methods That Don't Lie)
Standard A/B tests violate their core assumptions when applied to AI features. Here's how to measure real impact using causal inference methods that handle contamination, spillover, and long-horizon behavioral shifts.
The Cross-User Consistency Problem: When Your AI Gives Different Answers to the Same Question
Enterprise AI tools silently erode trust when teammates ask the same question and get different answers. Here's why temperature=0 doesn't fix it, and the engineering patterns that actually do.
The Dev-to-Prod Cost Shock: Why Your AI Feature Costs Pennies in Staging and Dollars in Production
Staging environments systematically hide the cost drivers that matter in production. Here's the gap between what you pay in dev and what hits your invoice at scale — and how to model it honestly.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
