Page 37
12 articlesThe Snapshot Trace Test: Production Traces as Your Regression Suite
Hand-curated LLM eval sets decay the moment user behavior shifts. Pin production traces, assert semantic equivalence on outputs, structural equality on tool calls, and latency bands instead of point estimates.
The Stop-Sequence Footgun: When User Input Collides With Your Delimiter
Stop sequences chosen for clean engineering examples become silent ambient hazards once user content joins the prompt. How the bug manifests, why eval suites miss it, and the reserved-namespace fix that prevents recurrence.
Streaming Structured Output: Why Your Parser Hangs on Token 47
Token-streaming and structured output are architecturally at odds. The naive try/catch JSON.parse loop is O(n²), the is_complete boolean is a lie, and partial enums are how a Delete tool fires on DeleteIfEmpty.
The Summary Tax: When Compaction Eats More Tokens Than It Saves
Long-running agents trigger summarization on overflow or hierarchically, and at scale the compaction passes quietly become the dominant inference cost — and the dashboard never tells you.
The Thumbs-Down on the Right Answer: When User Feedback Trains Sycophancy
Thumbs-down ratings mix wrongness with unwelcomeness. Optimizing prompts against the raw signal trains agreement, not accuracy — and the math gets worse with scale.
Token-Aware Logging: When Your Traces Cost More Than the Inference They Observe
Telemetry pipelines for AI agents now eat more budget than the LLM calls they observe. A field-by-field cost model — fingerprinted prompts, outcome-aware sampling, retention tiers — for keeping observability on the right side of the COGS line.
The Dependency Bomb in Your Tool Catalog: When Adding One Tool Breaks Five Agents
Adding a tool to your agent's catalog redistributes the planner's selection probability across every entry, silently re-routing workflows your eval suite never thought to test.
We Already Have That: When AI Features Reinvent Code You Already Own
Most AI features inside established companies duplicate logic the codebase already owns. The fix is an audit before the build, and a composition pattern that makes the model the fallback rather than the primary path.
The Attack Vector You Ship With Every Open RAG System
When users can contribute to your knowledge base, they're not the only ones writing to it. Five malicious documents in a 2.6M-entry corpus achieved a 97% attack success rate — and the pipeline showed no errors.
Fine-Tune Orphan: Recovering Domain Expertise When the Base Model Is Deprecated
When a base model is deprecated, fine-tuned domain expertise doesn't transfer automatically. Three recovery paths—behavioral distillation, re-labeling, and prompt encoding—and the preparation that makes the difference.
Statistical Watermarking for LLM Output: How Token Logit Bias Creates Detectable Signatures
LLM text watermarking embeds statistically detectable signatures in token logit probabilities at inference time. How green/red-list schemes work, why Google's SynthID-Text is semi-fragile, and what production engineers need to know before committing to watermarking for compliance or attribution.
The 80% Trap: How Aggregate RAG Metrics Hide Systematic Long-Tail Failures
RAG systems reporting 80% retrieval accuracy often hide systematic failures on tail queries. Here's how to audit coverage gaps and fix them without degrading head performance.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
