Page 26
12 articlesMobile App Store Review Meets AI Features: The Deploy Cadence Collision
Web AI features iterate in minutes; mobile AI features iterate on the platform's review clock. The architectural seams that keep both surfaces honest under one eval set and two release trains.
Model Migration Bills You Twice: The Eval Re-Anchoring Tax Nobody Prices
Swapping foundation models silently invalidates your eval baseline — human-anchored scores, LLM judges, snapshots, and team intuition all need re-anchoring, and the labor bill is usually larger than the per-token savings.
On-Call at 3am for an AI Feature That Didn't 500
The pager fires because eval-on-traffic dropped four points, not because a service crashed. Runbook patterns, alert design, and rotation discipline for the failure modes that don't trip your existing alerts.
Per-Customer Prompt Forks: Why Your Next Model Migration Is 47 Migrations
Per-customer system prompt customizations accumulate silently until model-migration day, when a single provider deprecation becomes 47 separate re-validations. The base-plus-overlay architecture and approval discipline that prevent it.
Pre-Commit Hooks for Prompts: The Inner-Loop Tooling LLM Teams Keep Shipping Without
Prompts now move more behavior than code, yet most teams review them with 2008-era tooling. Five pre-commit hooks — formatter, linter, secret-and-PII scanner, smoke eval, and cache-impact estimator — for treating prompt edits with the rigor they actually need.
The Prompt as Documentation: When the System Prompt Becomes the Only Artifact Anyone Trusts
When PMs, support, and sales start reading the system prompt to learn what the product does, it is both a flattering signal and a structural failure. Here is how to keep the part that works and fix the rest.
The Prompt Author Identity Problem: Three Roles Editing the Same File
Every production system prompt has three authors — engineering, product, and ML — and none of them agree on what a change is. Here's the structural fix.
The Prompt Graph Inside Your Agent: Cross-Prompt Regression Chains Nobody Mapped
A four-word planner edit moves the verifier's pass rate three points downstream. The fix is to treat your agent's prompt portfolio like a microservice mesh — graph, edges, contracts, blast-radius PR reviews, per-edge regression evals, edge owners.
Quarterly Model Migration: Make It a Calendar Event, Not a Fire Drill
Foundation-model providers retire models on a cadence your team did not plan for. Treating each migration as a one-off project pays the same setup tax three or four times a year. Run a quarterly drill instead — DRI, candidate model, regression rerun, runbook refresh — so the next deprecation email lands inside a rhythm the team already has.
RAG Against a Phantom Inventory: When Your Corpus Describes Features Your Product Removed
A RAG system retrieves docs about a feature you deleted four months ago and confidently walks the customer into a button that doesn't exist. The evals stay green. Here's why retrieval and grounding metrics miss this entire class of failure, and what has to change at the org level to fix it.
Rater Throughput Is the Hidden Bottleneck in Your Eval Pipeline
Rater throughput caps eval velocity in any AI system that takes human grading seriously. Here is the operational discipline — calibration cycles, queue-aware prioritization, rubric feedback loops — that treats annotation capacity as an SRE problem rather than a recruiting one.
Repeat-Question Detection: The Session-Level Blind Spot Your Per-Turn Eval Cannot See
Per-turn eval scores can stay green while users rephrase the same question three times and churn. The failure lives at the session level — and here is how to detect and grade it.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
