Page 98
12 articlesContext Poisoning in Long-Running AI Agents
How accumulated context in long-running AI agents silently corrupts reasoning, the four failure modes that cause it, and the checkpointing, pruning, and invariant-checking patterns that prevent cascading failures.
Debugging LLM Failures Systematically: A Field Guide for Engineers Who Can't Read Logs
When a prompt fails in production, most engineers cycle through random edits until something works. Here's the structured methodology — input ablation, boundary testing, intermediate inspection — that finds root causes in minutes instead of hours.
Document Injection: The Prompt Injection Vector Inside Every RAG Pipeline
Every PDF, Word doc, and spreadsheet your RAG pipeline ingests is a potential attack surface. Here's how document injection works, what it's already broken in production, and the sanitization architecture that actually defends against it.
Why Gradual Rollouts Don't Work for AI Features (And What to Do Instead)
Feature flags and canary deploys assume deterministic code. AI features are stochastic, quality degrades silently, and there's no real-time ground truth. Here's the mental model shift required to deploy AI safely.
The HITL Rubber Stamp Problem: Why Human-in-the-Loop Often Means Neither
Most human-in-the-loop implementations don't produce oversight — they produce paperwork. Here's why reviewers stop scrutinizing, and the design patterns that keep HITL meaningful at scale.
The Hybrid Automation Stack: A Decision Framework for Mixing Rules and LLMs
Rule-based automation is brittle but auditable. LLM automation is flexible but opaque. A practical decision framework for which tasks belong in which paradigm—and how to architect the seam between them.
Latency Budgets for AI Features: How to Set and Hit p95 SLOs When Your Core Component Is Stochastic
LLM latency doesn't behave like database latency. Here's how to define realistic p95 SLOs for AI-powered features, decompose the latency budget, and use hedging, streaming, and speculative execution to actually hit them.
The Provider Reliability Trap: Your LLM Vendor's SLA Is Now Your Users' SLA
You're advertising 99.9% uptime while your critical path runs through an API with a 99.5% SLA — and provider incidents cluster during peak traffic. Here's how to close the gap before an outage closes it for you.
LLMs as ETL Primitives: AI in the Data Pipeline, Not Just the Product
Most teams build AI features into their products. The quiet transformation is happening inside the data pipeline, where LLMs classify, enrich, deduplicate, and route records at scale — creating compounding data assets that product-only teams can't replicate.
The Multi-Tenant Prompt Problem: When One System Prompt Serves Many Masters
B2B AI products let customers customize behavior, but layered system prompts silently override each other — and nobody notices until an enterprise customer files a ticket. Here's the explicit instruction hierarchy that makes conflict resolution auditable.
The Multi-Variable Regression Problem: Isolating AI Failures When Everything Changed at Once
When your AI feature regresses and the model version, prompt, retrieval corpus, and tool schemas all changed on the same Friday, attribution becomes nearly impossible. Here's the controlled experiment discipline and shadow evaluation patterns that prevent the worst outcome.
The Operational Model Card: Deployment Documentation Labs Don't Publish
Published model cards tell you whether a model is safe — not whether it will hit your p95 SLA, what context lengths it degrades at, or how often it produces malformed JSON. Here's the test battery for building the deployment documentation you actually need.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
