Page 61
12 articlesThe Feedback Signal Timing Problem: Why Your AI Metrics Are Lying to You
Thumbs up/down and CSAT scores often predict the opposite of long-term AI product value. Here's how to build measurement systems that actually capture what matters.
Gating AI Features on Model Performance, Not User Segments
Traditional feature flags gate on user cohorts — but AI quality failures hit everyone simultaneously and never trigger error alerts. Here's how performance-conditioned gates fix that.
Gradual Context Replacement: Managing Long AI Conversations Without Losing Quality
Hard truncation and naive summarization both cause quality drops in long AI sessions. The rolling-replace pattern—keeping recent turns raw while compressing older ones incrementally—is the approach that holds quality as sessions scale past forty turns.
What Your Help Center Is Missing for AI Features (And Why Users Keep Filing Tickets)
Traditional how-to guides fail for AI features because they assume deterministic behavior. Here are the documentation formats — capability galleries, limitations sections, variance examples — that actually reduce support tickets.
The Helpful AI Paradox: Why Instruction-Following Is a Security Vulnerability
The same compliance that makes LLMs useful makes them exploitable. Here's the engineering reality behind prompt injection attacks, real-world breaches, and what defenses actually reduce risk.
HIPAA, SOC2, and Your Agent: The Architectural Constraints Compliance Actually Imposes
Most AI teams discover compliance requirements at audit time, not sprint one. Here's what HIPAA and SOC2 actually require architecturally — and the three decisions you cannot retrofit later.
Human Override as a First-Class Feature: Designing AI Systems That Fail Gracefully to Human Control
Most AI systems treat human takeover as an error state rather than a designed mode. Here's how to build override protocols that are first-class operational paths, not afterthoughts.
The Invisible Author Problem: Git Blame When AI Writes Most of Your Code
When 46% of code is AI-generated and carries no provenance metadata, git blame terminates at a developer who accepted a suggestion they may not have understood. Here's what breaks and what teams are doing about it.
LLM-as-Judge Adversarial Failures: When Your Eval Harness Gets Gamed
A null model with constant outputs topped AlpacaEval at 86.5% win rate. Here's how LLM judges get gamed, the structural biases they carry, and the audit protocol that keeps your eval pipeline honest.
Your Load Tests Are Lying: LLM Provider Capacity Contention in Production
LLM APIs are multi-tenant shared infrastructure — your load tests pass at 2 AM but production latency spikes at 9 AM Tuesday. Learn the mechanics of shared peak demand and the architectural patterns (multi-provider hedging, circuit breakers, reserved capacity) that protect your SLOs.
LLM Self-Debugging: When the Explanation Is the Signal vs. When It's the Lie
LLMs answer fluently when asked why they failed — but the explanation and the actual failure mechanism are often two different things. A practical guide to telling them apart before you act.
LLM Tail Latency: Why Your P99 Is a Disaster When P50 Looks Fine
LLM response time distributions are fundamentally heavy-tailed in ways that conventional API monitoring misses entirely. Here's how to diagnose the P99 gap and fix it.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
