Page 74
12 articlesEvaluating AI Service Vendors Beyond Your LLM Provider
Most teams scrutinize their LLM provider but trust everything else on vibes. A rigorous framework for evaluating guardrail vendors, embedding providers, observability tools, and fine-tuning platforms—with due diligence criteria that catch business-model risk before it bites you.
Foundation Model Vendor Strategy: What Enterprise SLAs Actually Guarantee
Enterprise teams pick LLM vendors based on benchmarks and demos. Then they hit production and discover what the SLA actually says — which is usually much less than they assumed.
The Evaluation Paradox: How Goodhart's Law Breaks AI Benchmarks
When AI teams optimize for benchmark scores instead of real capabilities, scores climb while quality degrades. Here's how the evaluation paradox works and what structural changes actually make evals resistant to gaming.
GraphRAG vs. Vector RAG: The Architecture Decision Teams Make Too Late
Vector RAG hits a mathematical ceiling on relational queries — the migration path from pure vector to hybrid graph-vector retrieval, and the query patterns that reveal you've outgrown dense-only search.
Hallucination Is Not a Root Cause: A Debugging Methodology for AI in Production
Moving beyond 'the model hallucinated' to systematic root cause analysis: retrieval failure, conflicting context, prompt ambiguity, and knowledge boundary violations each require different fixes.
Why Hallucination Rate Is the Wrong Primary Metric for Production LLM Systems
Hallucination rate is easy to measure but weakly correlated with user outcomes. A framework for choosing behavioral metrics that actually reflect whether your AI feature is working.
The Idempotency Problem in Agentic Tool Calling
Why agent retry logic causes duplicate charges, double-sent emails, and inconsistent state — and how saga patterns, idempotency keys, and structured error signals fix the problem at the architecture level.
The Inference Optimization Trap: Why Making One Model Faster Can Slow Down Your System
Swapping a model component for a faster version often increases end-to-end latency and cost. Here's why—and the profiling discipline that prevents it.
What Your Inference Provider Is Hiding From You: KV Cache, Batching, and the Latency Floor
The decisions made inside LLM inference infrastructure—KV cache eviction, continuous batching, chunked prefill—set your application's performance envelope before you write a line of code. Here's what's actually happening and the few knobs you control.
Invisible Model Drift: How Silent Provider Updates Break Production AI
LLM providers update models without changelogs. Your prompt regressions are real, they're silent, and they're your problem to detect. Here's how.
Knowledge Distillation for Production: Teaching Small Models to Do Big Model Tasks
How to use frontier model outputs as supervision signal to build task-specific small models—covering the dataset curation pipeline, quality collapse detection, and the benchmarking methodology that tells you when the distilled model is ready for production.
Knowledge Distillation Without Fine-Tuning: Extracting Frontier Model Capabilities Into Cheaper Inference Paths
A practical decision framework for AI engineers on when distilling frontier model capabilities into smaller student models actually pays off—and when it silently fails on out-of-distribution inputs.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
