Page 38
12 articlesThe Sparse Signal Problem: Measuring AI Feature Quality When You Can't A/B Test
B2B AI features rarely have enough daily users for A/B testing. Here's how to measure quality using Bayesian methods, proxy signals, and structured expert elicitation when frequentist statistics simply don't scale.
The Write Side of the Agent: Designing for Reversibility at the Action Layer
Every write operation an AI agent makes is a potential incident. How to design the action layer for reversibility before your agent deletes something you cannot get back.
The AI A/B Test That Lied: Novelty, Carryover, and Anchoring Bias in LLM Experiments
Three psychological biases systematically inflate AI feature A/B test results — novelty inflation, anchoring, and carryover — and the standard holdout group remedy fixes none of them. Here's the longitudinal cohort design that actually works.
AI Clarification Dialogues That Actually Converge: Designing for One-Turn Resolution
Multi-turn clarification loops frustrate users and degrade LLM performance. A framework for designing AI systems that resolve ambiguity in one turn using information-gain prioritization, confidence-threshold gating, and architectural constraints.
The AI Code Review Inversion: What to Focus on When the Author Is a Machine
When AI agents write the majority of your commits, line-by-line correctness review misses the bugs that matter. Here's the review discipline that actually works for machine-authored code.
AI Co-Pilot vs. AI Pilot: The Evidence-Based Product Decision Framework
The specific deployed-system signals — task completion rate, error recovery time, user override frequency, edge-case exposure — that determine whether AI should be advisory or autonomous, and why the wrong default costs you user trust that's hard to recover.
Why AI Quality Monitors Conflate Model Drift, Data Drift, and Prompt Drift — and What to Do About Each
When AI quality degrades in production, the root cause is one of three distinct problems — but conventional monitoring treats them all the same way and wastes weeks pointing at the wrong fix.
The AI Efficiency Paradox: When Your Best Feature Kills Your Revenue
AI features that make users more productive can compress per-seat revenue — a structural pricing problem that catches teams after the renewal cycle, not before. Here's how to think about it before you ship.
Story Points Don't Survive First Contact With an LLM
Why the assumptions behind velocity-based sprint planning collapse for AI features — and the milestone-based, eval-driven approach that keeps LLM engineering teams predictable.
AI Feature Dependency Graphs: Resilience Engineering When Your Services Share a Model
When fifteen product features share the same embedding model and LLM endpoint, one provider incident becomes a distributed systems outage with no stack trace. How to map AI feature dependencies, apply circuit breakers at each layer, and design degradation chains that fail features cleanly instead of corrupting outputs.
AI Feature PMF Signals: Why Your Metrics Are Lying to You
Conventional signals like NPS, thumbs-up ratings, and activation rates systematically mislead for AI features. Here's what genuine product-market fit actually looks like — and how to measure it.
Why Rolling Back an AI Feature Is Harder Than Rolling Back Code
A technical code rollback fixes the system, but it doesn't fix the users. Here's why AI behavior changes are sticky in ways code changes aren't, and the patterns that let you reclaim design space without breaking trust.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
