Page 33
12 articlesThe AI Feature Retirement Playbook: How to Sunset What Users Barely Adopted
Most engineering teams know how to ship AI features. Almost none have a plan for retiring them. Here's the playbook for knowing when to quit and how to do it without burning users or accumulating compliance debt.
The AI Incident Response Playbook: Diagnosing LLM Degradation in Production
When your LLM feature degrades in production, standard SRE runbooks leave you blind. Here's the diagnosis tree, prompt rollback strategy, and postmortem template built specifically for AI systems.
The AI Incident Runbook: When Your Agent Causes Real-World Harm
When an AI agent causes real-world harm, your existing outage runbook will mislead you. Here is the playbook built for stochastic systems: how to bound blast radius without stack traces, preserve evidence before it disappears, and investigate beyond 'the model hallucinated.'
The AI Output Copyright Trap: What Engineers Need to Know Before It's a Legal Problem
Training data memorization, derivative works doctrine, and output ownership are live legal disputes with direct engineering consequences. Here's the risk surface and the controls that actually reduce liability.
The AI Taste Problem: Measuring Quality When There's No Ground Truth
How to evaluate AI outputs when accuracy metrics are meaningless — the engineering discipline behind pairwise studies, inter-rater reliability, and LLM-as-judge for copywriting, creative content, and design.
The Three Silent Clocks of AI Technical Debt
AI-specific technical debt—prompt drift, eval erosion, and embedding staleness—compounds invisibly unlike code debt. Here's how to detect each clock before it runs down.
The Annotation Economy: Why Every Label Source Has a Hidden Tax
A decision framework for choosing between human domain experts, crowd workers, synthetic LLM generation, and behavioral inference for eval label sourcing—and when annotation-free is actually right.
Annotation-Free Evaluation: Measuring LLM Quality Before You Have Ground Truth
A practical guide to measuring LLM output quality in week one — before you have labeled data. Covers self-consistency, constraint satisfaction, behavioral invariants, and LLM-as-judge, with the failure modes of each.
The Feedback Loop You Never Closed: Turning User Behavior into AI Ground Truth
Explicit thumbs-up ratings are a lie. Edit rates, retry patterns, and session abandonment reveal far more about AI quality — and you can turn them into eval datasets without an annotation budget.
Benchmark Contamination: Why That 90% MMLU Score Doesn't Mean What You Think
Frontier models score impressively on standard benchmarks, but contamination — where test data leaks into pretraining — inflates those numbers significantly. Here's what the gap actually looks like and how to design evaluations that give honest signals.
Why 'Fix the Prompt' Is a Root Cause Fallacy: Blameless Postmortems for AI Systems
The 'fix the prompt' reflex is displacing real root cause analysis in AI incident postmortems. Here's why it happens and how to apply blameless SRE culture to non-deterministic systems.
Board-Level AI Governance: The Five Decisions Only Executives Can Make
Most AI governance writing targets MLOps teams. But five strategic decisions can only be made at the board level — and the regulatory exposure for getting them wrong is growing fast.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
