Page 117
12 articlesLLM Confidence Calibration in Production: Measuring and Fixing the Overconfidence Problem
LLMs that say 'I'm highly confident' are often wrong at that exact rate. How to measure calibration error, why RLHF makes it worse, and the production design patterns that actually help.
The Provider Abstraction Tax: Building LLM Applications That Can Swap Models Without Rewrites
Teams that build directly on one LLM provider accumulate prompt idioms, tool schema conventions, and behavioral dependencies that become migration debt. Here's the abstraction layer design that makes switching providers a configuration change rather than a multi-month rewrite.
LLMs in the Security Operations Center: Acceleration Without Liability
How to wire LLMs into security operations so they accelerate triage without quietly approving real intrusions — confidence thresholds, log-poisoning defenses, and the metrics that matter.
The max_tokens Knob Nobody Tunes: Output Truncation as a Cost Lever
Most teams pad max_tokens to avoid mid-generation cutoffs and pay for the slack forever. Per-route calibration against real output distributions can cut output token spend 20–40% without quality loss.
Your AI Feature Should Lose to a Regex First
Before you invest in fine-tuning or RAG, your AI feature should be required to beat the simplest deterministic baseline you can build. Most teams skip this gate and pay for it.
The Model EOL Clock: Treating Provider LLMs as External Dependencies
Every pinned model version has a deprecation date you don't control. Here's how to treat provider LLMs as external dependencies with behavioral regression suites, EOL runbooks, and migration test harnesses baked in before the notice arrives.
Model Routing Is a System Design Problem, Not a Config Option
Treating LLM selection as a runtime dispatch decision — not a deployment constant — unlocks real cost savings. Here's how to think about routing signals, fallback failure modes, shadow routing, and the cost accounting that most teams skip.
Multi-Model Consistency: When Your Pipeline's Sequential LLM Calls Contradict Each Other
Three LLM calls in a single workflow can produce conflicting facts, entity references, and state claims. Here's how to design pipelines that stay coherent.
Multi-Session Eval Design: Catching the AI Feature That Gets Worse Over Time
Single-turn evals miss the class of AI failure that emerges only after state accumulates. How to design a multi-session eval harness, decay curves, and regression methodology that catch quality rot before users churn.
Multi-User Shared Agent State: The Concurrency Primitives You Actually Need
Most agent designs assume one user per session. Shared workspaces need distributed systems primitives to prevent silent data corruption when concurrent users give contradictory instructions.
Multimodal Pipelines in Production: What Breaks When You Go Beyond Text
Going multimodal in production means confronting a new class of failures: silent image rejections, PDF table misalignment, audio latency budgets, and cross-modal hallucination that text evals never surface.
The Noisy Neighbor Problem in Shared LLM Infrastructure: Tenancy Models for AI Features
When one feature's batch job eats the shared API quota, paying users see 429s. Detection signals and isolation patterns for shared LLM infrastructure.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
