Page 122
12 articlesLLMs as ETL Primitives: AI in the Data Pipeline, Not Just the Product
Most teams build AI features into their products. The quiet transformation is happening inside the data pipeline, where LLMs classify, enrich, deduplicate, and route records at scale — creating compounding data assets that product-only teams can't replicate.
The Multi-Tenant Prompt Problem: When One System Prompt Serves Many Masters
B2B AI products let customers customize behavior, but layered system prompts silently override each other — and nobody notices until an enterprise customer files a ticket. Here's the explicit instruction hierarchy that makes conflict resolution auditable.
The Multi-Variable Regression Problem: Isolating AI Failures When Everything Changed at Once
When your AI feature regresses and the model version, prompt, retrieval corpus, and tool schemas all changed on the same Friday, attribution becomes nearly impossible. Here's the controlled experiment discipline and shadow evaluation patterns that prevent the worst outcome.
The Operational Model Card: Deployment Documentation Labs Don't Publish
Published model cards tell you whether a model is safe — not whether it will hit your p95 SLA, what context lengths it degrades at, or how often it produces malformed JSON. Here's the test battery for building the deployment documentation you actually need.
Prompt Linting: The Pre-Deployment Gate Your AI System Is Missing
Most teams ship prompt changes to production with less scrutiny than a CSS tweak. Static analysis for prompts — catching conflicting instructions, injection-vulnerable template slots, and positional traps — is the pre-deployment gate your AI system is missing.
Schema Entropy: Why Your Tool Definitions Are Rotting in Production
Tool definitions in production AI systems degrade silently over months. Here's how schema entropy forms, why agents can't self-correct, and the versioning and contract-testing practices that catch rot before it breaks live agents.
The Selective Abstention Problem: Why AI Systems That Always Answer Are Broken
Most AI product design optimizes for better answers. The harder, more valuable capability is principled non-answering — and almost no team builds it deliberately.
The Semantic Validation Layer: Why JSON Schema Isn't Enough for Production LLM Outputs
Constrained decoding guarantees your LLM outputs are valid JSON. It cannot guarantee they make sense. Here's the two-layer validation architecture that catches the failures schema can't see.
Silent Async Agent Failures: Why Your AI Jobs Die Without Anyone Noticing
Async AI jobs fail silently and confidently — HTTP 200, dashboards green, customers eventually complaining. Here's how dead letter queues, idempotency keys, and saga logs translate from conventional distributed systems to fix the problem.
Staffing AI Engineering Teams: Who Owns What When Every Feature Has an AI Component
How the skills split between ML engineers, data engineers, and product engineers shifts when LLMs commoditize modeling—and how to staff, structure, and assign ownership when every feature has an AI component.
Stale Retrieval: The Data Quality Problem Your RAG Pipeline Is Hiding
RAG pipelines fail silently when their retrieval corpus drifts — outdated facts, deleted documents, and stale embeddings that pass every faithfulness metric. Here's how to detect it, propagate deletions, and build freshness into your pipeline from the start.
Your LLM Eval Is Lying to You: The Statistical Power Problem
Most LLM eval suites run on 50–200 examples and claim significance they don't have. Here's the math that shows why your evals can't detect the improvements you're making — and what to do about it.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
