Page 34
12 articlesThe Disable Switch Is the Real Product: Designing the Non-AI Fallback Path
Most AI features ship with a disable switch that returns an error. Treat the off-state as a real product, with capability-level flags, deterministic fallbacks, and the same eval discipline as the on-state.
Disconnected Agent Mode: Designing for the Network You Don't Have
Most AI agent stacks assume the network is always there. That assumption breaks on planes, in basements, and behind flaky Wi-Fi. Here is how disconnected-first architecture actually works.
Distillation Is a Product Decision, Not a Research Artifact
Distillation is a product decision about which capabilities you sacrifice to unlock a cost floor and a latency floor — not a research-team optimization. A frontier-model feature and its distilled variant are two products, not two implementations of one.
Personalization Belongs in a Dotfile, Not a Vector Store
Most agent personalization is tone, formatting, default tools, and project context — declarative settings the dotfile pattern solved decades ago. Reach for fine-tuning and persistent memory only after you have exhausted config.
Eval-as-Code: When Your Release Gate Is a Notebook on Someone's Laptop
The number deciding whether your model ships is a notebook on someone's laptop. Treat the eval suite like the production system it has quietly become — version it, gate it, give it SLOs.
The Eval Automation Trap: When Your Pipeline Drifts Away From What Users Actually Want
Automated eval pipelines can show improving accuracy while user satisfaction quietly drops. Here's how the drift happens and how to catch it before it compounds.
The Eval Migration Tax: Why a Prompt Schema Change Wrecks 800 Test Cases
Schema changes to AI prompts routinely break hundreds of test cases that have nothing to do with the change. Most teams treat eval suites as static fixtures instead of versioned data — and pay a hidden tax on every release.
The Fallback Cascade: Why Your AI Feature Needs Five Failure Modes, Not One
AI features that treat model failures as binary pass/fail events are one outage away from disaster. A five-level fallback cascade — from frontier model to human escalation — keeps your feature functional when individual levels break.
The First 100 Tickets After You Launch an AI Feature
The eight-week sequence of operational tickets every AI feature launch produces — cost spikes, eval drift, latency tails, silent provider updates — and the launch playbook that pre-stages the answers.
The Human Review Queue Is Your P0 SLA: When HITL Becomes the Bottleneck
The human-in-the-loop escalation path you wired up for safety three months ago is now the silent bottleneck of your AI feature. Here is how to treat it as a production system with its own SLOs, capacity model, and feedback loop — before customers tell you first.
The IDE Plugin Is the Product Now: When Your Coding Agent Outgrows the Editor's Plugin API
Why the leading AI coding tools forked the editor instead of staying as plugins, and how to decide between extending VS Code, forking it, or building from scratch.
The LLM-as-Validator Antipattern: Why Your AI Quality Gate Has a Blind Spot
Using an LLM to evaluate LLM outputs as your primary quality gate creates a circular validation loop blind to systematic model failures. Here's what to use instead.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
