Page 28
12 articlesAI Code Review Drift: When Your LLM Reviewer's Standards Mutate Faster Than the Code
An LLM code reviewer is not a stable tool — it's a stack of independently drifting components. Here's why your PR bot's catch rate decays silently and what calibration discipline keeps the safety net from thinning.
AI Feature Dependency Graphs: When a Prompt Edit Is a Silent Breaking Change
A prompt edit is a breaking change to every downstream feature that consumes the output. Manifests, live-corpus contract tests, and drift alerts are how teams draw the AI dependency graph before the next outage draws it for them.
Annotation Drift: How Your Eval Set Stops Measuring the Product You Ship
An eval score that climbs while the product silently decays is a measurement system whose calibration has slipped. Here is how annotation drift hides in plain sight, why both the rubric and the product move under your feet, and the four moves that keep eval numbers honest.
Asymmetric Eval Economics: Why One Eval Case Costs More Than the Feature It Tests
A single eval case routinely costs more engineering effort than the feature it tests. Why teams underinvest in evals, and why the capex frame fixes it.
Background Agents and the Notification Budget: Why Proactive AI Hits a Hard Ceiling at User Attention
Proactive AI agents collide with a hard daily ceiling of three to five notifications per user. Teams that don't budget attention ship features whose launch metric inverts their retention metric within weeks.
Conversation History Is a Trust Boundary, Not a Text Blob
Conversation history is a multi-source feed, not append-only state. Tag each turn's origin, anchor user turns with HMACs, and wrap tool output in trust zones — or your agent's attack surface grows linearly with every turn.
The Demo-to-Dogfood Gap: Why Your AI Feature Dies Between the Launch Slide and Monday Morning
Most enterprise AI pilots leave a great demo and a dead Slack channel. The dogfood phase is the cheapest production-grade eval you will ever run — here is what a real gate looks like and why the demo is not evidence of readiness.
The Embedding Migration Black Hole: How a Vector Model Bump Silently Rewrites Your Business Rules
An embedding model upgrade is sold as an infra swap but ships as a recalibration event. Here's the parallel system of thresholds, clusters, and gold labels you have to rebuild — and the migration plan that survives production.
The Eval Backfill Tax: Why Every Model Capability Launch Costs More Than You Budgeted
New model capabilities introduce failure modes your historical eval suite was never designed to catch — and the work to backfill it is the unbudgeted critical path on every capability launch.
The Eval Bus Factor: When the Person Who Defined 'Correct' Walks Out the Door
Eval suites stay green long after the person who knew what they were testing has left. The damage is silent, the recovery is expensive, and the fix is organizational, not technical.
Eval Triage Queues: Why FIFO Misses the Failures That Matter
A FIFO queue of eval failures wastes the most expensive thing in the loop — reviewer time. Score failures by traffic, severity, and recency, batch by cluster, and protect an adversarial quota.
The MCP Capability Disclosure Tax: When Every Connected Server Bills Your Context Window
MCP tool definitions reload on every planning turn, quietly burning 15-66K tokens per call and degrading tool-selection accuracy as servers stack. Here's how to price the disclosure tax and contain it with progressive disclosure, per-server attribution, and stable schemas.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
