Page 21
12 articlesThe Quadratic Cost of a Conversation: Why AI Chat Spend Grows Superlinearly
A ten-turn chat costs about 55x a single turn, not 10x, because every turn re-bills the whole history. Here is the N-squared math, why caching does not fix it, and how to bound it.
The Rate Limit That Became a Product Decision
Provider quotas no longer stay inside the backend. When an agent hits a tokens-per-minute ceiling mid-task, the failure lands on the user — so the rate limit is now something product has to design around.
The Semantic Cache That Confidently Returns the Wrong Answer
A semantic cache trades the exact-match guarantee for latency, and the bill is a false hit served as fluent fact. How to measure the false-hit rate, pick the embedding model, cache the retrieval not the generation, and tune the threshold as a safety decision.
Shadow AI: The Agents Your Team Already Shipped
Half your workforce is already running unapproved AI tools, and a crackdown only pushes them out of sight. Why shadow AI is a symptom of a slow official path — and how a faster paved road fixes it.
The Streaming Rollback Problem: You Can't Un-Say a Token
Streaming output reaches the user before any guardrail can check it. Why output moderation and token streaming are in structural conflict, and how to shrink the exposure window instead of pretending it away.
The Streaming Token the User Acted On Too Soon
Token-by-token streaming makes assistants feel fast, but it also exposes the model's unfinished thinking as a finished answer. Here is the race condition that causes it and the design patterns that fix it.
Structured Output Is Not Validated Output
JSON mode and constrained decoding guarantee the shape of an LLM response, not its meaning. Why a passing schema check is the start of correctness work, and where semantic validation actually belongs.
Your System Prompt Grows After Every Incident — and Nobody Deletes a Line
Every production incident leaves a defensive sentence in your system prompt, and nobody ever deletes one. Here is why prompt accretion is real technical debt and how to prune it with dating, half-lives, and ablation.
Task Completion Goes Green While Users Quietly Suffer
A 94% task-completion dashboard can stay green while an agent burns tokens, backtracks, and exhausts users. Why completion is the wrong number and four trajectory metrics that see what it cannot.
The Test the Agent Wrote That Tests Nothing
Coding agents generate test suites that pass, raise coverage, and catch nothing. Why agent-written tests drift into tautologies, and how mutation testing and red-green discipline make them constrain behavior again.
When Your Test Set Leaks Into Fine-Tuning: The Contamination You Cause Yourself
Benchmark contamination is usually blamed on model vendors, but the worst leaks are the ones your own team creates — failure triage, synthetic data, and shared RAG corpora that quietly move eval cases into training.
The Agent That Remembers What You Took Back: Deletion as a First-Class Memory Operation
An append-only agent memory store rots the moment a stored fact becomes false. Why deletion, retraction, and invalidation must be first-class operations — and how to design memory writes that can be found, contradicted, and removed.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
