Page 28
12 articlesThe LLM Judge Is a Versioned Dependency, Not Neutral Infrastructure
Grade model outputs with an LLM judge and the judge is a model with its own behavior. The day it changes, every historical score becomes a foreign currency — and most teams never notice.
When LLMs Review LLMs, Errors Get Laundered Not Caught
A closed loop where one model reviews another and feeds the next eval has no ground truth anywhere — errors get laundered into high scores. Here is where to put the human back.
The Model Reached End of Life and Took Your Prompt With It
A model deprecation notice reads like a one-line config change, but the prompt you tuned for six months was fitted to one model's quirks and does not survive the swap. Treat model end-of-life as a recurring migration project with a re-runnable eval set.
Onboarding an Agent Like a Junior Engineer Is a Category Error
A junior engineer accumulates context every week; an agent accumulates nothing. Why the new-hire metaphor misallocates your attention, and where to put the learning instead.
The Permission Prompt Is a UX Bug: When Human-in-the-Loop Becomes Human-as-Rubber-Stamp
Agent products gate dangerous actions behind approval dialogs and call it oversight, but by the fortieth prompt the human clicks approve on reflex. Why prompt volume is the real safety bug, and how to fix it.
Prompt Caching's Hidden Tax: When a Cache Hit Serves the Wrong User's Context
A prompt cache key is a correctness boundary, not a billing knob. Draw it for hit rate and you invite cross-tenant context bleed and stale personalization.
Prompt Injection Is a Confused Deputy, Not a Content-Filtering Problem
Treating prompt injection as a content-filtering problem is a losing arms race. The real vulnerability is a confused deputy: an agent acting on untrusted instructions with borrowed authority. Scope the capability instead.
The Quadratic Cost of a Conversation: Why AI Chat Spend Grows Superlinearly
A ten-turn chat costs about 55x a single turn, not 10x, because every turn re-bills the whole history. Here is the N-squared math, why caching does not fix it, and how to bound it.
The Rate Limit That Became a Product Decision
Provider quotas no longer stay inside the backend. When an agent hits a tokens-per-minute ceiling mid-task, the failure lands on the user — so the rate limit is now something product has to design around.
The Semantic Cache That Confidently Returns the Wrong Answer
A semantic cache trades the exact-match guarantee for latency, and the bill is a false hit served as fluent fact. How to measure the false-hit rate, pick the embedding model, cache the retrieval not the generation, and tune the threshold as a safety decision.
Shadow AI: The Agents Your Team Already Shipped
Half your workforce is already running unapproved AI tools, and a crackdown only pushes them out of sight. Why shadow AI is a symptom of a slow official path — and how a faster paved road fixes it.
The Streaming Rollback Problem: You Can't Un-Say a Token
Streaming output reaches the user before any guardrail can check it. Why output moderation and token streaming are in structural conflict, and how to shrink the exposure window instead of pretending it away.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
