Page 22
12 articlesYour A/B Test Can't Tell Two Models Apart When the Metric Is Vibes
Click-through rate cannot distinguish a model your users love from one they tolerate. Before you trust an experiment to pick a model, prove the metric can detect a model you broke on purpose.
Your Agent's Audit Log Records Everything Except the Reason
A complete agent trace shows what happened and never why. Why observability is not explainability, why a logged chain-of-thought can be fiction, and how to capture decision rationale that survives a regulator.
The Chatbot That Inherited Your Support Team's Worst Habits
Fine-tuning on real customer-service transcripts transfers your team's tacit workflows along with the domain knowledge. Here is what your model actually learned, and the curation and evals that catch it.
Your Embeddings Don't Know the Contractor Was Off-Boarded
Your vector index is a permission cache nobody is invalidating. When access changes at the source, the embeddings keep answering as if nothing happened — and that is the leak nobody plans for.
Your Eval Set Only Has Problems You Already Solved
Your LLM eval score is climbing because survivorship bias filters out the users who never came back. Here is how to find the failures your suite cannot see.
From a Bug to a Behavior Rate: The AI Postmortem Without a Reproducer
An AI incident with no reproducer is not a debugging failure — it is the system telling you that one bad output is a sample from a distribution, not a deterministic bug. The postmortem has to change shape.
Your Model Router Is a Load Balancer That Cannot See the Load
A model router decides which model handles a query before the model has done anything — the difficulty signal it needs only exists in the answer. Why classifier accuracy lies, why misroutes look like mediocre quality instead of errors, and how to instrument the downstream signals that actually move with routing quality.
A Prompt Diff Hides Its Own Blast Radius
A three-word prompt edit and a three-paragraph rewrite look identical in a text diff, yet their behavioral consequences are not. Why prompt review needs eval deltas, not character counts.
Re-Ask Rate: The Failure Signal Your Eval Pipeline Never Extracts
Users who repeat the same question in a session are telling you the previous answer failed — but turn-level evals and end-of-session CSAT both miss it. How to instrument re-ask rate as a first-class metric.
Shadow Replay Punishes the Model That Would Have Changed the Conversation
Shadow replay evaluations quietly punish better models by scoring them against logged user turns that were conditioned on the old model — here is why, and what replay can still measure honestly.
Streamed Tokens Are a Promise You Can't Take Back
A confident token-streamed answer collapses when its backing tool call fails. Streaming is an irreversibility contract — and there are patterns that buy back optionality without giving up perceived latency.
The Agent That Read Last Week's Slack Like It Was Yesterday
An agent retrieves a six-week-old message that says 'we'll ship tomorrow' and treats it as a present-tense plan. The retrieval pipeline kept the body and threw away the clock.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
