Page 114
12 articlesGPU Memory Math for Multi-Model Serving: Why Most Teams Over-Provision by 3x
KV cache, not model weights, dominates GPU memory under concurrent load. The exact formulas for capacity planning, quantization tradeoffs (AWQ vs GPTQ vs GGUF), and bin-packing strategies that let you serve 4 models on hardware budgeted for 1.
Graph Memory for LLM Agents: The Relational Blind Spots That Flat Vectors Miss
Vector search retrieves similar facts but can't recover how facts relate — the structural blind spot that breaks agents handling multi-hop queries, evolving state, and long-horizon reasoning. Here's what graph memory fixes and what it costs.
Building a Hallucination Detection Pipeline for Production LLMs
A three-stage pipeline combining sentinel classification, token-level detection, and NLI verification catches LLM fabrications, contradictions, and outdated claims under 200ms P99 latency in production.
The Hidden Scratchpad Problem: Why Output Monitoring Alone Can't Secure Production AI Agents
Frontier models acknowledge the influence of sensitive inputs in their visible reasoning only 25–41% of the time. Here's why output-layer monitoring can't secure production agents—and how to build oversight that accounts for hidden computation.
The Hidden Token Tax: Where 30-60% of Your Context Window Disappears Before Users Say a Word
System prompts, tool schemas, chat history, and safety preambles silently consume 30-60% of your LLM context window before user content arrives — here's how to audit and reclaim it.
Hybrid Cloud-Edge LLM Architecture: Routing Inference Where It Actually Belongs
70-80% of production LLM queries never need a frontier model. A hybrid cloud-edge architecture routes each request to the cheapest tier that handles it well — using complexity classifiers, confidence cascading, and speculative decoding to cut costs 50-100x on the edge path without sacrificing quality.
Hybrid Cloud-Edge LLM Inference: The Routing Layer That Determines Your Cost, Latency, and Privacy Profile
A routing layer between edge and cloud inference cuts LLM costs 60–80% while improving latency and privacy — here's the engineering behind query-level routing, model compression, speculative decoding, and the orchestration that makes hybrid architectures work in production.
Hybrid Cloud-Edge LLM Inference: The Latency-Privacy-Cost Triangle That Determines Where Your Model Runs
A production guide to splitting LLM inference between on-device models and cloud APIs — covering the latency-privacy-cost triangle, compression techniques that preserve task accuracy, intelligent query routing, and the failure modes unique to hybrid architectures.
Hybrid Cloud-Edge LLM Inference: When On-Device Models Beat the Cloud
Production teams are routing 60–80% of LLM queries to on-device models — cutting latency below 20ms, eliminating data-residency headaches, and slashing cloud inference costs. A practical guide to the routing, compression, and architecture patterns behind hybrid cloud-edge inference.
How to Integration-Test AI Agent Workflows in CI Without Mocking the Model Away
A three-tier CI testing architecture for AI agents that avoids both the cost of live API calls and the hollowness of mocking the model away — using StubLLM test doubles, VCR cassette replay, and tool contract tests to catch orchestration bugs before they reach production.
The Intent Gap: When Your LLM Answers the Wrong Question Perfectly
Intent misalignment causes 32% of dissatisfactory LLM responses — models answer the literal question while missing what the user actually needed. Here's why it evades your evals and how to close the gap.
LLM Queuing Theory: Why Your Load Balancer Thinks in Requests While Your GPU Thinks in Tokens
Applying Little's Law, priority queuing, and admission control to token-based LLM inference workloads — why request-level load balancing fails, how work-conserving schedulers unlock 30-70% more GPU throughput, and the capacity planning math that prevents production surprises.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
