Page 66
12 articlesPrompt Deprecation Contracts: Why a Wording Cleanup Is a Breaking Change
A four-word edit to a system prompt can break parsers, judges, and chained agents that pinned the old wording. Prompts are APIs with silent consumers — and the discipline that keeps them stable looks a lot like REST endpoint deprecation.
Prompt Linting Is the Missing Layer Between Eval and Production
Behavioral evals catch what your model says; they cannot catch what your prompt is. A prompt linter — fast, deterministic, structural — closes the gap that ships eval-green and surfaces as a 11pm production incident.
Prompt Position Is Policy: The Silent Merge Conflict When Three Teams Co-Own a System Prompt
When a system prompt grows past 2K tokens, position bias makes a moved instruction as load-bearing as a rewritten one — and line-based diffs hide it. How three teams silently overwrite each other's intent, and the section-ownership and eval discipline that catches it.
The Customer Record Hiding in Your Few-Shot Prompt Template
The 'representative customer' you pasted into your few-shot prompt six months ago is still in production — re-identifiable, re-shipped, and invisible to DLP.
The Refusal Latency Tax: Why Layered Guardrails Eat Your p95 Budget
Layered safety pipelines silently triple p95 latency and cost on the long tail. Treat guardrails as a budgeted resource with tiered classifiers, parallel checks, and an honest latency contract.
The Reranker Is the Silent Second Model Your RAG Eval Never Measures
Most RAG pipelines run two models in series — retriever and reranker — but eval suites only grade the generator's output. When the reranker drifts, the dashboard shows answer quality dropping with no causal arrow. Here's how to build a reranker eval that catches the silent regressions.
Retiring an AI Feature Is a Trust Event, Not a Deprecation
Sunsetting an AI assistant breaks differently than deprecating an API — the playbook needs cohort cuts, a maintenance-cost ledger, and comms calibrated to relationships, not contracts.
Retries Aren't Free: The FinOps Math of LLM Retry Policies
Classical retry policies assume bounded cost and independent retries. LLM workloads break both — and the bill compounds on the worst inputs. A field guide to rebuilding retry budgets for token economics.
Retrieval Sprawl: When 'Just Add RAG' Becomes the Architectural Diversion
Adding a retrieval step to fix every model failure looks like progress until your system is a pile of retrievers gluing together a prompt that still has the original problem. A diagnostic framework, ablation discipline, and complexity budget for RAG.
Your Review Queue Is Where the Autonomy Promise Goes to Die
Human-in-the-loop AI fails quietly: the review queue grows, latency creeps, and the safety story breaks one item at a time. A field guide to SLOs, capacity tripwires, and tiered review for AI features.
The Same Prompt at 3 PM and 3 AM Is Not the Same Prompt: Diurnal Drift in LLM Evaluation
An LLM call's behavior depends on the wall clock — batch size, cache state, and routing tier shift with provider load. Evals that run at 2 AM calibrate on conditions production never sees. Five practices that close the gap between off-peak eval and peak-hour reality.
The 70% Reliability Uncanny Valley: Where AI Features Go to Lose User Trust
An AI feature that succeeds 70% of the time can be worse than one that fails 70% of the time — concentrated, unpredictable failures collapse user trust faster than consistent unreliability. Why aggregate accuracy lies, why users cannot self-calibrate, and how to design for the uncanny zone.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
