Page 146
12 articlesCompensating Transactions and Failure Recovery for Agentic Systems
AI agents fail mid-workflow. Here's how to apply the saga pattern, idempotency keys, and durable checkpointing so irreversible tool calls — emails, charges, deletions — can be recovered without manual intervention.
Temporal Reasoning Failures in Production AI Systems
LLMs confidently answer questions about 'current' events using training data that may be 12–30 months stale. Here's how staleness differs from hallucination, why you can't prompt-engineer your way out of it, and what to actually do about it in production.
Why Your Agent UI Feels Broken (And How to Fix It)
Most agent UIs fail not because the model is bad, but because the interaction layer is broken. A practical breakdown of the five root causes and the engineering patterns that fix them.
Red-Teaming AI Agents: The Adversarial Testing Methodology That Finds Real Failures
A practical methodology for red-teaming AI agents in production — covering goal hijacking, tool-level attacks, multi-agent exploitation, memory poisoning, and why aggregate metrics hide the failures that matter most.
Prompt Versioning and Change Management in Production AI Systems
When a three-word prompt change breaks a revenue pipeline with no rollback path, the root cause is always the same: prompts treated as ephemeral config rather than software. A complete guide to prompt versioning, silent regression detection, canary rollouts, and the organizational problem of who owns prompt changes.
Test-Driven Development for LLM Applications: Where the Analogy Holds and Where It Breaks
Most teams write evals after prompts — which guarantees weak evals. Here's how eval-first development works, and the four places where the TDD analogy breaks down for LLMs.
LLM API Resilience in Production: Rate Limits, Failover, and the Hidden Costs of Naive Retry Logic
LLM providers run at 99–99.5% uptime — 6–14x worse than cloud infrastructure. Here's the minimum viable resilience stack: jitter, circuit breakers, dual rate limiting, and multi-provider failover that actually works in production.
LLM Latency Decomposition: Why TTFT and Throughput Are Different Problems
TTFT and throughput are not two ends of a slider — they're caused by different physics and require different fixes. A practical guide to decomposing LLM latency and optimizing the right metric for your workload.
Agent Sandboxing and Secure Code Execution: Matching Isolation Depth to Risk
A practical guide to the agent sandbox spectrum — from Docker containers to Firecracker microVMs — covering capability restriction models, real-world escape vectors, and a decision framework for matching isolation depth to risk.
Async Agent Workflows: Designing for Long-Running Tasks
Bridging the 30-second HTTP wall: async job queues, idempotency keys, checkpoint-resume patterns, and polling vs. webhooks for AI agents that run for minutes or hours.
Designing Approval Gates for Autonomous AI Agents
Engineering patterns for pausing AI agent execution at the right moments — action risk classification, interrupt-checkpoint-resume, async approval workflows, and circuit breakers that prevent silent drift.
MCP in Production: What Nobody Tells You About the Model Context Protocol
MCP grew 8,000% in five months, but most teams shipped without understanding the latency traps, security vulnerabilities, and architectural anti-patterns that surface at scale. A practical guide for engineers running MCP in production.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
