Page 3
12 articlesThe Distillation That Lost a Capability Your Eval Suite Never Measured
Distillation optimizes a divergence over a finite sample, then ships against a finite eval. Behaviors the eval never measured are free entropy the student is licensed to drop — and the ones it drops first are usually the rare-but-load-bearing ones.
The Embedding Model Rotation That Shadowed Your A/B Test for a Quarter
Why vendor-side embedding upgrades silently break your A/B tests on retrieval features, and the experimentation discipline that closes the gap.
The Escalation Path That Routes Back to the Agent
An escalate_to_human tool stops being human-in-the-loop the moment the downstream queue grows its own automation. Why the contract has to outlive the consumer.
The Eval Harness Whose Judge Model Was Upgraded Silently
An LLM judge whose endpoint silently updates is a measurement instrument with no calibration contract. Pin snapshots, build anchor sets, and run dual-judge windows so a six-point lift means your system improved — not the ruler.
The Eval Rubric Pulled By Two Drift Vectors
An eval rubric read by humans and an LLM judge drifts on two axes at once. Composite scores hide the motion. Here is the measurement protocol that keeps each drift attributable.
The Eval Set That Sampled Production Traffic at 3am EST
An offline eval built from a nightly 3am cron quietly becomes a survey of overnight batch retries and APAC traffic — and the leaderboard cannot tell you whose model it is.
The Eval That Converges, Then Quietly Collapses
A plateaued eval score does not always mean a model ceiling. When labelers homogenize, agreement metrics climb and the eval stops measuring what the team thinks it does.
The Feature Flag Your Model Already Learned to Predict From the Inputs It Could See
An LLM prompt experiment leaks assignment whenever the routing hash and the prompt assembler share an input — a walk through how the lift gets manufactured, the symptoms your dashboard does not surface, and the disciplines that close the gap.
The Fine-Tune Cold Start Your Provider Bills as Idle Time
Hosted fine-tunes share an API surface with base models but not a cost-of-latency curve. Here's why the cold start tax hides in your p99 and never shows up on the bill.
The Fine-Tune Dataset You Accidentally Built While Debugging
When the thumbs-down button in your staging UI silently doubles as a training pipeline, you fine-tune on six months of personal taste, customer text, and engineer venting. Separate the debug surface from the curation surface — or ship a model trained on whatever your team was feeling that week.
The Fine-Tune That Erased the Alignment You Inherited
Supervised fine-tuning quietly strips the refusal training your base model came with. Why task-only evals miss it, and the four practices that catch the regression before customers do.
The GPU Reservation Your Batch Workload Starved Your Real-Time Path On
Sharing one GPU pool between nightly training and morning inference looks like a utilization win until the p99 dashboard prices the externality. Why GPU partitions have to be physical, accounting has to follow latency class, and the morning tail won't be fixed in software.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
