Page 70
12 articlesAI Product Metrics That Don't Lie: Behavioral Signals Over Thumbs-Up Scores
Aggregate satisfaction scores and thumbs-up rates hide the cases where AI is confidently wrong. Here's the behavioral signal stack that actually tells you whether your model improvement moved the needle.
The AI Reliability Floor: Why 80% Accurate Is Worse Than No AI at All
There is a reliability floor below which an AI feature actively destroys user trust faster than it can build value. Here is how to find it before shipping.
The AI Procurement Gap: Why Your Vendor Evaluation Process Can't Handle Probabilistic Systems
Traditional RFPs score features and uptime SLAs that mean nothing for stochastic outputs. The eval-driven assessment, contract clauses, and vendor transparency signals that procurement teams are missing for AI.
Stop Writing Prompts by Hand: Automated Optimization with DSPy and MIPRO
DSPy and its MIPRO optimizer replace manual prompt engineering with declarative signatures and Bayesian search — producing prompts that outperform hand-written ones by 20–40% on complex tasks. Here's how the system works and when it's worth the overhead.
Backpressure for LLM Pipelines: Queue Theory Applied to Token-Based Services
How to apply Little's Law, admission control, bulkheads, and token-bucket backpressure to LLM call graphs — and why naive retry logic turns transient provider blips into outages.
The Bias Audit You Keep Skipping: Engineering Demographic Fairness into Your LLM Pipeline
Safety filters and fairness checks are different problems requiring different engineering responses. Output quality disparities across gender, race, and language group won't surface in your guardrails — here's the methodology that catches them before they ship.
The Cognitive Offloading Trap: When Your Team Can't Work Without the AI
Engineering teams that route all knowledge work through AI agents stop practicing the underlying skills. Here's how to recognize unhealthy AI dependency and design deliberate practices that preserve human capability.
Compound Failure Modes in AI Pipelines: When Partial Success Isn't Enough
If each stage of your AI pipeline succeeds 95% of the time, a three-step chain succeeds only 86% of the time. The probability math practitioners undercount, correlation effects that make it dramatically worse, and the architectural patterns that prevent multiplicative collapse in production.
Context Compression Changes What Your Model Actually Sees
Token pruning and prompt compression can cut LLM inference costs by 3–10x, but they silently change what your model sees. A practical breakdown of the failure modes — lost coreference chains, dropped constraints, tool output hallucination — and how to validate and budget compression safely.
Continuous Fine-Tuning Without Data Contamination: The Production Pipeline
A production engineering guide to ongoing LLM fine-tuning from user feedback — covering data routing architecture, contamination detection, catastrophic forgetting prevention, and automated safety preservation.
Contract Tests for Prompts: Stop One Team's Edit From Breaking Another Team's Agent
Prompts are shared APIs without contracts — a consumer-driven testing discipline catches cross-team breaking changes before they hit production agents.
Data Quality Gates for Agentic Write Paths: Garbage In, Irreversible Actions Out
Agents with write-access tools translate upstream data quality failures directly into real-world side effects. Here's the validation architecture that prevents them.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
