Page 53
12 articlesThe Evaluation Paradox: How Goodhart's Law Breaks AI Benchmarks
When AI teams optimize for benchmark scores instead of real capabilities, scores climb while quality degrades. Here's how the evaluation paradox works and what structural changes actually make evals resistant to gaming.
GraphRAG vs. Vector RAG: The Architecture Decision Teams Make Too Late
Vector RAG hits a mathematical ceiling on relational queries — the migration path from pure vector to hybrid graph-vector retrieval, and the query patterns that reveal you've outgrown dense-only search.
Hallucination Is Not a Root Cause: A Debugging Methodology for AI in Production
Moving beyond 'the model hallucinated' to systematic root cause analysis: retrieval failure, conflicting context, prompt ambiguity, and knowledge boundary violations each require different fixes.
Why Hallucination Rate Is the Wrong Primary Metric for Production LLM Systems
Hallucination rate is easy to measure but weakly correlated with user outcomes. A framework for choosing behavioral metrics that actually reflect whether your AI feature is working.
The Idempotency Problem in Agentic Tool Calling
Why agent retry logic causes duplicate charges, double-sent emails, and inconsistent state — and how saga patterns, idempotency keys, and structured error signals fix the problem at the architecture level.
The Inference Optimization Trap: Why Making One Model Faster Can Slow Down Your System
Swapping a model component for a faster version often increases end-to-end latency and cost. Here's why—and the profiling discipline that prevents it.
What Your Inference Provider Is Hiding From You: KV Cache, Batching, and the Latency Floor
The decisions made inside LLM inference infrastructure—KV cache eviction, continuous batching, chunked prefill—set your application's performance envelope before you write a line of code. Here's what's actually happening and the few knobs you control.
Invisible Model Drift: How Silent Provider Updates Break Production AI
LLM providers update models without changelogs. Your prompt regressions are real, they're silent, and they're your problem to detect. Here's how.
Knowledge Distillation for Production: Teaching Small Models to Do Big Model Tasks
How to use frontier model outputs as supervision signal to build task-specific small models—covering the dataset curation pipeline, quality collapse detection, and the benchmarking methodology that tells you when the distilled model is ready for production.
Knowledge Distillation Without Fine-Tuning: Extracting Frontier Model Capabilities Into Cheaper Inference Paths
A practical decision framework for AI engineers on when distilling frontier model capabilities into smaller student models actually pays off—and when it silently fails on out-of-distribution inputs.
The Latent Capability Ceiling: When a Bigger Model Won't Fix Your Problem
Frontier models plateau on domain-specific tasks well before teams expect it. Here's how to diagnose whether you've hit a true capability ceiling or a prompt, eval, or data problem — and which technique actually breaks through.
The Idempotency Crisis: LLM Agents as Event Stream Consumers
At-least-once delivery assumes reprocessing an event produces the same result. LLMs don't. A practical guide to idempotency keys, deduplication windows, and compensating read-models for AI-powered Kafka consumers.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
