Page 99
12 articlesPrompt Linting: The Pre-Deployment Gate Your AI System Is Missing
Most teams ship prompt changes to production with less scrutiny than a CSS tweak. Static analysis for prompts — catching conflicting instructions, injection-vulnerable template slots, and positional traps — is the pre-deployment gate your AI system is missing.
Schema Entropy: Why Your Tool Definitions Are Rotting in Production
Tool definitions in production AI systems degrade silently over months. Here's how schema entropy forms, why agents can't self-correct, and the versioning and contract-testing practices that catch rot before it breaks live agents.
The Selective Abstention Problem: Why AI Systems That Always Answer Are Broken
Most AI product design optimizes for better answers. The harder, more valuable capability is principled non-answering — and almost no team builds it deliberately.
The Semantic Validation Layer: Why JSON Schema Isn't Enough for Production LLM Outputs
Constrained decoding guarantees your LLM outputs are valid JSON. It cannot guarantee they make sense. Here's the two-layer validation architecture that catches the failures schema can't see.
Silent Async Agent Failures: Why Your AI Jobs Die Without Anyone Noticing
Async AI jobs fail silently and confidently — HTTP 200, dashboards green, customers eventually complaining. Here's how dead letter queues, idempotency keys, and saga logs translate from conventional distributed systems to fix the problem.
Staffing AI Engineering Teams: Who Owns What When Every Feature Has an AI Component
How the skills split between ML engineers, data engineers, and product engineers shifts when LLMs commoditize modeling—and how to staff, structure, and assign ownership when every feature has an AI component.
Stale Retrieval: The Data Quality Problem Your RAG Pipeline Is Hiding
RAG pipelines fail silently when their retrieval corpus drifts — outdated facts, deleted documents, and stale embeddings that pass every faithfulness metric. Here's how to detect it, propagate deletions, and build freshness into your pipeline from the start.
Your LLM Eval Is Lying to You: The Statistical Power Problem
Most LLM eval suites run on 50–200 examples and claim significance they don't have. Here's the math that shows why your evals can't detect the improvements you're making — and what to do about it.
The AI Adoption Paradox: Why the Highest-Value Domains Get AI Last
Healthcare sits at 39% AI adoption while software companies hit 92% — yet healthcare has more to gain. The gap isn't risk aversion. It's a structural mismatch between accuracy thresholds, compliance timing, and deployment architecture.
The AI Rollback Ritual: Post-Incident Recovery When the Damage Is Behavioral, Not Binary
Behavioral regressions in LLM systems don't fail your tests or trigger your alerts. Here's how to detect, diagnose, and recover from the failure mode that looks like success.
The Curriculum Trap: Why Fine-Tuning on Your Best Examples Produces Mediocre Models
Curating only high-quality, confident outputs as fine-tuning data creates distribution mismatch, destroys uncertainty awareness, and produces models that are confidently wrong. Here's why—and what to do instead.
The Integration Test Mirage: Why Mocked Tool Outputs Hide Your Agent's Real Failure Modes
Agents built against mocks never encounter the failures that bite in production: pagination loops, rate limits mid-sequence, partial success responses, and schema ambiguity. Here's what to do instead.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
