Page 73
12 articlesPrompt Bisect: Binary-Searching the Edit That Broke Your Eval
An overnight two-point eval drop and a prompt PR with seventeen edits is a binary search problem, not a guessing game. Here is how to bisect a prompt the way kernel maintainers bisect a kernel — and the commit-granularity discipline it forces on the team.
Prompt-Eligibility: The Missing Column in Your Data Classification
Most data classification schemes never modeled the prompt layer as a vendor egress channel. Adding a prompt-eligibility tier — and the template audit that fills it — closes a compliance gap your DLP scheme silently denies.
Your System Prompt Will Leak: Designing for Prompt Extraction
Prompt extraction is the quiet attack on LLM products. Treat the system prompt as public, move secrets out of context, and build an eval for it.
Prompt-Version Skew Across Regions: The Unintended A/B Test Your CDN Ran for Six Hours
Pushing prompts through CDN-style rollout systems creates silent geography-split A/B tests when one region drifts ahead of another. Here is the rollout discipline, observability dimension, and rollback model that keep prompt versions globally coherent.
When Your CLI Speaks English: Least Authority for Promptable Infrastructure
The principle of least authority breaks when your CLI accepts English. Every wrapper that translates intent into commands becomes a confused deputy. Patterns that hold up: intent-bound tokens that pin the resolved plan, binding dry-runs, and audit trails that link prompt to action graph.
The Indexing Policy Committee Nobody Convened: RAG Corpus Governance Beyond the One-Time Migration
Most RAG failures aren't model failures—they're governance failures. The four axes of corpus governance—legal, freshness, authorship-trust, ownership—and the indexing-policy discipline that determines whether your retrieval index is a product surface or a shared inbox.
The RAG Read-After-Write Race: When Your Vector Index Cites a Document That No Longer Exists
Production RAG pipelines silently assume snapshot isolation between retrieval and generation. They never enforce it — and the bug shows up as deleted-chunk citations, edited-chunk inversions, and stale-permission leaks.
Reachability Analysis for Agent Action Spaces: Eval Coverage for the Branches You Never Tested
Your tool catalog plus a planner forms a reachable graph of plans your evals have probably never exercised. Borrow reachability analysis from compilers to find the branches your incident channel will discover first.
Reasoning-Effort Budgeting: When Thinking Tokens Become a Finance Line Item
Reasoning tokens look like output tokens on the bill but balloon 3–10x and have no natural ceiling. Treat thinking effort as a tunable resource — measured in yield, governed by budgets, routed by difficulty, and surfaced as its own dashboard line item before finance asks about it first.
Replan, Don't Retry: Why Most Agent Errors Aren't Transient
Most agent frameworks default to exponential-backoff retry on tool errors — a pattern borrowed from stateless HTTP that's actively wrong inside a stateful planning loop. The right default is replan.
Reviewing Agent PRs Is a Different Job, Not a Faster One
Agent-authored PRs concentrate bugs in different places than human PRs, and the reviewer instincts trained on years of human code quietly fail on them. A walkthrough of the new bug profile, why fluent diffs are dangerous, and the three artifacts every reviewer now has to read together.
The RLAIF Doom Loop: When Your Cheapest Feedback Signal Quietly Poisons Your Fine-Tune
AI-generated preference labels are 100x cheaper than human ones — and they teach your model to prefer the judge's aesthetic, not your users'.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
