Page 2
12 articlesThe Conversation Summary Your Agent Regenerated Each Turn Because the Cache Key Included a Timestamp
A one-line debugging change added a timestamp to a summary cache key and silently tripled the LLM bill for two weeks — a study in why cache keys are a contract, not plumbing, and why hit rate must be a first-class signal.
The Cost Dashboard Your Finance Team Built That Excluded the Embeddings Re-index
Per-feature dashboards track token spend. Per-provider dashboards track invoices. The quarterly embeddings re-index falls between both and lands in an unowned infrastructure bucket — where 40% of AI spend goes to die unreviewed.
The Data Residency Contract Your Provider Honored at the API Boundary and Broke at the Cache
Regional API endpoints commit to where your request goes, not where the cached prefix bytes that satisfy it live. The auditable boundary and the cache placement boundary are governed by different SLAs — and the gap is where compliance posture breaks.
The Debug Logger That Put Your System Prompt in a Customer-Readable Audit Feed
A field your platform team added for incident triage ended up in a tenant audit export — the disclosure required no attacker, just two correct decisions composing into a leak.
The Deprecation Date That Moved While It Sat in Your Backlog
A provider can amend a deprecation date in place with no diff and no notification. The team that filed the original date in the deferred bucket finds out the way support tickets find them.
The Downstream API That Kept Writing After the User Cancelled the Conversation
Hitting stop closes the LLM stream cleanly. It does not stop the HTTP request the tool already opened to a third party that has no idea the conversation ended. Here is why AbortSignal stops at the socket, and what to build at the commit boundary instead.
The Embedding Deprecation That Halved Your Retrieval Recall Without a Deploy
A deprecated embedding endpoint that quietly routes to a 'compatibility' successor can halve your retrieval recall without a deploy. Here's why query/document embedding mismatch is the silent killer of RAG, and how to pin endpoints to the corpus they produced.
The Eval Harness That Ran on Yesterday's Prompt Template After Your Team Shipped a New One
An eval suite that grades the wrong prompt version reports green on a broken release. The fix is not faster cache invalidation — it is content-addressed prompt hashes that make eval/prod drift impossible to express.
The Eval Rubric That Weighted Tone Next to Correctness and Quietly Selected Against Being Right
An equal-weight composite of helpfulness, clarity, empathy, and accuracy will quietly reward hedged wrongness over blunt correctness. Here is why the dashboard goes green while the product regresses, and the rubric patterns that put the gradient back where you wanted it.
The Eval Set Your Prompt Engineers Turned Into Production Few-Shots
A team-level data leak hides inside an improving eval dashboard when prompt engineers reuse curated eval examples as few-shot demonstrations. Here is why the contamination is invisible, what independence actually requires, and who has to be empowered to say no.
The Fallback Model Whose System Prompt Was Tuned for Someone Else
Failover keeps your LLM app available when the primary goes down — but the fallback model reads a system prompt that was tuned for someone else, and your users notice the difference.
The Few-Shot Example Your Model Treated as Binding Precedent
Few-shot examples are not neutral demonstrations — they are case law. Models bind to the closest example by surface tokens and inherit its constraints, shipping confident-wrong answers eval suites cannot see.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
