Page 122
12 articlesPrompt Caching: The Optimization That Cuts LLM Costs by 90%
A practical guide to cutting LLM API costs by 60–90% through prompt caching — covering prefix caching for Anthropic and OpenAI, the parallel execution trap that silently kills hit rates, and a multi-tier caching architecture for production workloads.
LLM Observability in Production: The Four Silent Failures Engineers Miss
Four failure modes that standard monitoring misses in production LLM systems — and how distributed tracing, continuous evaluation, and the right telemetry schema let you catch them before users do.
Prompt Injection in Production: The Attack Patterns That Actually Work and How to Stop Them
Prompt injection is the #1 LLM vulnerability — and most teams' defenses fail against adaptive attackers. A practical guide to the attack patterns causing real CVEs and the architectural controls that actually reduce risk.
LLM Routing: How to Stop Paying Frontier Model Prices for Simple Queries
Sending every query to your most expensive model is costing you 27x more than necessary. A practical guide to LLM routing strategies — rule-based, classifier, and cascade — with real benchmark numbers and the failure modes that will bite you.
Token Budget Strategies for Production LLM Applications
A practical guide to managing token budgets in production LLM systems — covering context rot, tiered allocation, summarization, KV cache exploitation, and the middleware layer that prevents silent agent failures.
Speculative Execution in AI Pipelines: Cutting Latency by Betting on the Future
Most LLM pipelines are sequential by accident. Speculative execution — running parallel hypotheses, pre-fetching tool calls, and generating candidate outputs simultaneously — can cut perceived latency 2–4×, but only when you understand when the coordination overhead cancels the gains.
Load Testing LLM Applications: Why k6 and Locust Lie to You
Conventional load testing tools measure the wrong things for LLM APIs. Learn which metrics actually matter—TTFT, inter-token latency, goodput—and how to build tests that predict production behavior instead of hiding its failure modes.
The Eval-to-Production Gap: Why 92% on Your Test Suite Means 40% User Satisfaction
High eval scores and low user satisfaction often coexist — here's why curated test sets diverge from real traffic and the four instrumentation changes that actually close the gap.
Compensating Transactions and Failure Recovery for Agentic Systems
AI agents fail mid-workflow. Here's how to apply the saga pattern, idempotency keys, and durable checkpointing so irreversible tool calls — emails, charges, deletions — can be recovered without manual intervention.
Temporal Reasoning Failures in Production AI Systems
LLMs confidently answer questions about 'current' events using training data that may be 12–30 months stale. Here's how staleness differs from hallucination, why you can't prompt-engineer your way out of it, and what to actually do about it in production.
Why Your Agent UI Feels Broken (And How to Fix It)
Most agent UIs fail not because the model is bad, but because the interaction layer is broken. A practical breakdown of the five root causes and the engineering patterns that fix them.
Red-Teaming AI Agents: The Adversarial Testing Methodology That Finds Real Failures
A practical methodology for red-teaming AI agents in production — covering goal hijacking, tool-level attacks, multi-agent exploitation, memory poisoning, and why aggregate metrics hide the failures that matter most.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
