Page 60
12 articlesWriting Acceptance Criteria for Non-Deterministic AI Features
Standard acceptance criteria break when your system is probabilistic. Here are the eval threshold contracts, example-based specs, and measurement patterns that let product and engineering agree on 'done' for AI features.
Tracing the Planning Layer: Why Your Agent Traces Are Missing Half the Story
Agent observability tools give you complete tool-call logs and timing, but the planning and reasoning that drove those decisions stays invisible. Here's what planning-layer tracing looks like, why it catches a completely different failure class, and how to instrument it today.
Agentic Web Data Extraction at Scale: When Agents Replace Scrapers
AI agents solve real problems traditional scrapers can't, but the 'LLM reads the page' prototype collapses at 1,000 pages per hour. Here's the hybrid architecture, cost model, and monitoring design that actually works in production.
The Accessibility Gap in AI Interfaces Nobody Is Shipping Around
Streaming token-by-token output breaks screen readers in ways most teams never test. Here's why WCAG has no answer for it, and the design patterns that actually work.
AI Agents in Your CI Pipeline: How to Gate Deployments That Can't Be Unit Tested
Traditional CI/CD infrastructure wasn't designed for non-deterministic software. Here's how to add meaningful deployment gates for LLM-powered features without turning your pipeline into a money-burning eval farm.
The Silent Regression: How to Communicate AI Behavioral Changes Without Losing User Trust
When you silently update a model or prompt, power users experience real regression even when aggregate metrics improve. Here's how to detect behavioral drift and communicate AI changes without destroying user trust.
The Debugging Regression: How AI-Generated Code Shifts the Incident-Response Cost Curve
AI code generation delivers real upfront velocity, but the cost appears downstream — at 3am, when the engineer on-call lacks the mental model to debug code they didn't write and barely reviewed.
AI Code Review at Scale: When Your Bot Creates More Work Than It Saves
The false-positive math that determines whether an AI PR reviewer accelerates or exhausts your team, what issue categories AI reviewers catch reliably vs. miss, and how to measure whether your code review agent is net positive.
AI-Assisted Codebase Migration at Scale: Automating the Upgrades Nobody Wants to Touch
How AI agents handle bulk code migrations—deprecated APIs, framework upgrades, language version evolution—where the wins are massive, where they create more work than they save, and the verification strategy that makes either approach safe.
The AI Engineering Career Ladder: Why Your SWE Leveling Framework Is Lying to You
Standard SWE leveling frameworks systematically misread AI engineer performance. Here's what actually distinguishes junior from senior when models do most of the coding.
The AI-Everywhere Antipattern: When Adding LLMs Makes Your Pipeline Worse
Adding an LLM to every step of your pipeline is the fastest way to make it slower, more expensive, and harder to debug. Here's the decision framework for knowing when AI genuinely helps versus when a lookup table is the right answer.
1% Error Rate, 10 Million Users: The Math of AI Failures at Scale
Why accuracy metrics that look fine in offline evals become catastrophic at production volume, how to set SLOs for AI features that account for tail behavior, and the product decision of what to do when a model is good enough but still wrong millions of times per month.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
