Page 138
12 articlesHybrid Cloud-Edge LLM Inference: When On-Device Models Beat the Cloud
Production teams are routing 60–80% of LLM queries to on-device models — cutting latency below 20ms, eliminating data-residency headaches, and slashing cloud inference costs. A practical guide to the routing, compression, and architecture patterns behind hybrid cloud-edge inference.
How to Integration-Test AI Agent Workflows in CI Without Mocking the Model Away
A three-tier CI testing architecture for AI agents that avoids both the cost of live API calls and the hollowness of mocking the model away — using StubLLM test doubles, VCR cassette replay, and tool contract tests to catch orchestration bugs before they reach production.
The Intent Gap: When Your LLM Answers the Wrong Question Perfectly
Intent misalignment causes 32% of dissatisfactory LLM responses — models answer the literal question while missing what the user actually needed. Here's why it evades your evals and how to close the gap.
LLM Queuing Theory: Why Your Load Balancer Thinks in Requests While Your GPU Thinks in Tokens
Applying Little's Law, priority queuing, and admission control to token-based LLM inference workloads — why request-level load balancing fails, how work-conserving schedulers unlock 30-70% more GPU throughput, and the capacity planning math that prevents production surprises.
The LLM Request Lifecycle Is a State Machine — Treat It Like One
LLM requests aren't linear—they silently traverse retry, fallback, and validation states that most teams never instrument. Modeling the request lifecycle as an explicit finite state machine makes every transition visible, debuggable, and cost-attributable.
The LLM Request Lifecycle Your try/catch Is Missing
Wrapping LLM calls in try/catch only catches the easy failures. A state machine approach makes retry, fallback, validation, and escalation paths first-class observable states — and surfaces the failure modes that return HTTP 200.
The Long-Horizon Evaluation Gap: Why Your Agent Passes Every Benchmark and Still Fails in Production
Single-turn benchmarks give a false sense of security for production AI agents. A model scoring 75% on SWE-Bench Verified collapses to under 25% on real engineering tasks—here's why the gap is structural and how to build evals that catch it.
MCP Server Supply Chain Risk: When Your Agent's Tools Become Attack Vectors
Third-party MCP servers are the new npm left-pad problem for AI agents. Real breaches — from Postmark email exfiltration to mcp-remote command injection — reveal five attack vectors and the layered defense patterns that reduce exposure without killing composability.
MoE Models in Production: The Serving Quirks Dense-Model Benchmarks Hide
Sparse MoE models need 8.6× more GPU memory than their active-parameter count implies, exhibit latency variance that dense-model monitoring misses, and break naive batching assumptions. Here's the serving analysis that benchmarks skip.
Model Fingerprinting: Detecting Silent Provider-Side LLM Swaps Before They Wreck Your Evals
When your LLM provider silently updates the model behind a stable API endpoint, your evals keep passing while your users notice the difference. Here's the fingerprinting and drift-detection stack that catches it first.
The Model Migration Playbook: How to Swap Foundation Models Without Breaking Production
A step-by-step playbook for safely migrating foundation models in production — shadow testing, embedding reindexing, prompt adaptation, canary rollouts, and the organizational coordination that separates a two-week swap from a two-month one.
The Model Migration Playbook: How to Swap Foundation Models Without a Feature Freeze
A phased production playbook for swapping LLM foundation models — covering shadow deployments, prompt re-engineering across providers, embedding reindexing strategies, and why your eval suite alone won't catch the regressions that matter.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
