Page 14
12 articlesThe Thumbs-Down on the Right Answer: When User Feedback Trains Sycophancy
Thumbs-down ratings mix wrongness with unwelcomeness. Optimizing prompts against the raw signal trains agreement, not accuracy — and the math gets worse with scale.
Token-Aware Logging: When Your Traces Cost More Than the Inference They Observe
Telemetry pipelines for AI agents now eat more budget than the LLM calls they observe. A field-by-field cost model — fingerprinted prompts, outcome-aware sampling, retention tiers — for keeping observability on the right side of the COGS line.
The Dependency Bomb in Your Tool Catalog: When Adding One Tool Breaks Five Agents
Adding a tool to your agent's catalog redistributes the planner's selection probability across every entry, silently re-routing workflows your eval suite never thought to test.
We Already Have That: When AI Features Reinvent Code You Already Own
Most AI features inside established companies duplicate logic the codebase already owns. The fix is an audit before the build, and a composition pattern that makes the model the fallback rather than the primary path.
The Attack Vector You Ship With Every Open RAG System
When users can contribute to your knowledge base, they're not the only ones writing to it. Five malicious documents in a 2.6M-entry corpus achieved a 97% attack success rate — and the pipeline showed no errors.
Fine-Tune Orphan: Recovering Domain Expertise When the Base Model Is Deprecated
When a base model is deprecated, fine-tuned domain expertise doesn't transfer automatically. Three recovery paths—behavioral distillation, re-labeling, and prompt encoding—and the preparation that makes the difference.
Statistical Watermarking for LLM Output: How Token Logit Bias Creates Detectable Signatures
LLM text watermarking embeds statistically detectable signatures in token logit probabilities at inference time. How green/red-list schemes work, why Google's SynthID-Text is semi-fragile, and what production engineers need to know before committing to watermarking for compliance or attribution.
The 80% Trap: How Aggregate RAG Metrics Hide Systematic Long-Tail Failures
RAG systems reporting 80% retrieval accuracy often hide systematic failures on tail queries. Here's how to audit coverage gaps and fix them without degrading head performance.
The Sparse Signal Problem: Measuring AI Feature Quality When You Can't A/B Test
B2B AI features rarely have enough daily users for A/B testing. Here's how to measure quality using Bayesian methods, proxy signals, and structured expert elicitation when frequentist statistics simply don't scale.
The Write Side of the Agent: Designing for Reversibility at the Action Layer
Every write operation an AI agent makes is a potential incident. How to design the action layer for reversibility before your agent deletes something you cannot get back.
The AI A/B Test That Lied: Novelty, Carryover, and Anchoring Bias in LLM Experiments
Three psychological biases systematically inflate AI feature A/B test results — novelty inflation, anchoring, and carryover — and the standard holdout group remedy fixes none of them. Here's the longitudinal cohort design that actually works.
AI Clarification Dialogues That Actually Converge: Designing for One-Turn Resolution
Multi-turn clarification loops frustrate users and degrade LLM performance. A framework for designing AI systems that resolve ambiguity in one turn using information-gain prioritization, confidence-threshold gating, and architectural constraints.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
