Page 83
12 articlesPattern-Matching Failures: When Your LLM Solves the Wrong Problem Fluently
Fluent, on-topic LLM answers that solve the wrong problem are the hardest bug class in production. A practical playbook for detecting surface-feature overfitting and designing prompts that expose it.
Plan-and-Execute Is Marketing, Not Contract: Plan Adherence as a First-Class SLI
Plan-and-execute agents emit plans that look like contracts but behave like forecasts. Treat plan adherence as an SLI with measurement, enforcement, and bounded re-planning budgets — not a quality nice-to-have you grade once a quarter.
Your Planner Knows About Tools Your User Can't Call
Scoping the tools list at execution time is too late. If the planner sees the full catalog, its refusals, clarifying questions, and reasoning trace leak capability existence to users who aren't authorized to know.
Popularity Bias in Vector Retrieval: Why the Same Five Chunks Dominate Every Query
Why a few chunks dominate every RAG query — how high-dimensional hubness and ANN graph structure silently collapse retrieval diversity, and the diagnostics plus mitigations that keep the long tail alive.
The Prompt Ownership Problem: When Conway's Law Comes for Your Prompts
Prompts live in four teams at once — authors, evaluators, deployers, and support. When no single role owns the whole loop, Conway's law guarantees silent quality leaks. The RACI gaps, shared-library traps, and steward role that actually keep behavior coherent.
Your Prompt Is Competing With What the Model Already Knows
Foundation models arrive pre-loaded with strong opinions about your domain. Probe the prior, refute the default, and stop shipping prompts that compete with what the model already believes.
Your RAG Chunker Is a Database Schema Nobody Code-Reviewed
Treat your RAG chunker like preprocessing and every boundary tweak becomes a silent schema migration. Version it, stage it, and own the retrieval eval alongside it.
Why Your RAG Citations Are Lying: Post-Hoc Rationalization in Source Attribution
Between 50 and 90 percent of LLM citations do not fully support the claims they are attached to. Here is why post-hoc attribution makes RAG systems quietly untrustworthy, how to measure citation faithfulness with NLI, and the architectural fixes that actually help.
Rate Limit Hierarchy Collapse: When Your Agent Loop DoSes Itself
One user's agent fan-out can starve every other user of the same quota. Why flat token buckets collapse under agent workloads, and the four-layer hierarchy that keeps the platform honest.
The Reasoning-Model Tax at Tool Boundaries
Reasoning models win benchmarks but bleed latency and quality at tool-choice steps. A per-step hybrid routing pattern, attribution, and anti-patterns.
The Reflection Placebo: Why Plan-Reflect-Replan Loops Return Version One
Single-model reflection loops mostly return the first plan with cosmetic edits while compounding the token bill. Here is how to measure the placebo and what actually produces divergent plans.
The Refusal Training Gap: Why Your Model Says No to the Wrong Questions
Refusal in language models is two distinct capabilities that training pipelines conflate, leaving models that block benign requests while confidently fabricating answers to questions they cannot reliably answer.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
