Page 43
12 articlesWhy Hallucination Rate Is the Wrong Primary Metric for Production LLM Systems
Hallucination rate is easy to measure but weakly correlated with user outcomes. A framework for choosing behavioral metrics that actually reflect whether your AI feature is working.
The Idempotency Problem in Agentic Tool Calling
Why agent retry logic causes duplicate charges, double-sent emails, and inconsistent state — and how saga patterns, idempotency keys, and structured error signals fix the problem at the architecture level.
The Inference Optimization Trap: Why Making One Model Faster Can Slow Down Your System
Swapping a model component for a faster version often increases end-to-end latency and cost. Here's why—and the profiling discipline that prevents it.
What Your Inference Provider Is Hiding From You: KV Cache, Batching, and the Latency Floor
The decisions made inside LLM inference infrastructure—KV cache eviction, continuous batching, chunked prefill—set your application's performance envelope before you write a line of code. Here's what's actually happening and the few knobs you control.
Invisible Model Drift: How Silent Provider Updates Break Production AI
LLM providers update models without changelogs. Your prompt regressions are real, they're silent, and they're your problem to detect. Here's how.
Knowledge Distillation for Production: Teaching Small Models to Do Big Model Tasks
How to use frontier model outputs as supervision signal to build task-specific small models—covering the dataset curation pipeline, quality collapse detection, and the benchmarking methodology that tells you when the distilled model is ready for production.
Knowledge Distillation Without Fine-Tuning: Extracting Frontier Model Capabilities Into Cheaper Inference Paths
A practical decision framework for AI engineers on when distilling frontier model capabilities into smaller student models actually pays off—and when it silently fails on out-of-distribution inputs.
The Latent Capability Ceiling: When a Bigger Model Won't Fix Your Problem
Frontier models plateau on domain-specific tasks well before teams expect it. Here's how to diagnose whether you've hit a true capability ceiling or a prompt, eval, or data problem — and which technique actually breaks through.
The Idempotency Crisis: LLM Agents as Event Stream Consumers
At-least-once delivery assumes reprocessing an event produces the same result. LLMs don't. A practical guide to idempotency keys, deduplication windows, and compensating read-models for AI-powered Kafka consumers.
LLM-Powered Data Pipelines: The ETL Tier Nobody Benchmarks
Most LLM benchmarks measure chatbot quality. But the bulk of enterprise LLM spend is going into batch pipelines — and almost nobody is measuring whether those pipelines actually work.
LLM Vendor Lock-In Is a Spectrum, Not a Binary
Not all LLM dependencies are created equal. Some are acceptable engineering tradeoffs; others are technical debt from day one. Here's how to tell them apart across six distinct lock-in layers.
Long-Session Context Degradation: How Multi-Turn Conversations Go Stale
Sessions beyond 50 turns accumulate contradictions, user intent drift, and sycophancy loops. Here's the engineering playbook for detecting degradation and keeping long conversations useful.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
