Page 36
12 articlesThe Phantom Skill: When Your Agent Demonstrates Capabilities You Never Tested For
When users build workflows on AI agent behaviors no test ever verified, you're shipping capabilities you cannot defend. A discipline for finding phantom skills before the next model upgrade silently removes them.
The Policy File: Why Your Refusal Rules Don't Belong in Your System Prompt
Production system prompts are three config files in a trench coat — conversational voice, output formatting, and refusal policy crammed into one artifact with one reviewer and one release cadence. Every policy edit becomes a behavioral regression on unrelated tasks. Here is the factoring that pays for itself.
Production Bias Auditing: Catching AI Discrimination Before Your Users Do
Pre-launch fairness audits expire the moment a model meets real traffic. A practical playbook for the metrics, slice-level audits, regression gates, and monitoring infrastructure that catch AI bias drift before it reaches users.
Prompt Edits Aren't Wording Changes: A Code Review Discipline for Prompts as Software
Prompt edits look like English but behave like code. The review discipline — paired eval-and-prompt PRs, behavioral diff comments, split reviewer roles — that catches behavioral regressions before users do.
Provider-Side Safety Drift: When Your Product Regresses Without a Deploy
Pinning a model ID does not freeze behavior. Refusal thresholds and content classifiers move server-side without a release note, and the regression is asymmetric across the safety boundary.
The Freshness-Relevance Tradeoff in RAG: Why You Can't Optimize Both at Query Time
Pure semantic retrieval ignores time, while recency-weighted retrieval rewards activity over correctness. A practical look at per-query time-sensitivity classifiers, per-document volatility scores, and the two-axis scoring that makes RAG correct on both stable and time-varying questions.
Reading the Agent Stack Trace: Triangulating Failures Across Model, Tool, and Harness
Most agent bugs live in the joints between model, tools, and harness — single-layer logs cannot see them. Build a unified trace, an OpenTelemetry GenAI span surface, a cause-hypothesis panel, and a reproducibility envelope to debug agents like the distributed systems they are.
The Refusal Audit: Why a Single Refusal Rate Hides Half the Failure Distribution
Refusal rate is a two-sided distribution, but most safety dashboards plot only one side. Here is what to instrument, how to sample, and who should own the calibration.
Retrieval Cascade Failure: How Document Deletion Poisons Your RAG Pipeline
When source documents disappear, their embeddings linger in the vector index and keep returning confidently wrong answers. A field guide to tombstones, cascade invalidation, and retrieval-time freshness checks.
The Session Boundary Problem: Where a Conversation Ends for Billing, Eval, and Memory
One session_id column, three meanings — billing, eval, and memory each define a conversation differently, and a single default ships three unrelated bugs with the same root cause.
The Show Your Work UX Trap: When the Reasoning Trace Is Debug Output Wearing a Product Costume
Most AI features ship with a visible reasoning trace because the model emits one and hiding it feels wasteful. It is a product decision the team never made — and a measurable source of trust loss.
Smaller Model, Bigger Bill: Why Cheaper-Per-Token Often Costs More
Switching to a smaller model to cut cost-per-token can quietly raise your LLM bill. The right unit is cost per successful task, and most dashboards never measure it.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
