Page 65
12 articlesThe LLM-Judge Ceiling: Why Your Auto-Eval Stops Correlating With Users at the Score That Matters
LLM-as-judge agreement with humans is highest in the muddy middle and collapses at the decision boundary. The discipline that keeps the unlock honest: per-slice kappa, drift dashboards, cross-family ensembles for high-stakes slices, and an explicit ceiling past which humans grade.
The LLM SDK Upgrade Tax: Why a Patch Bump Is a Model Rollout in Disguise
A patch bump on your model SDK can quietly rewrite prompt behavior, break JSON parsing, and ship regressions past your eval gate. Here is the discipline that catches it.
Your APM Is Quietly Dropping LLM Telemetry, and the Bug Lives in the Gap
Traditional APMs were built for bounded dimensions and stateless services. LLM workloads have a cardinality profile closer to product analytics, and the mismatch silently strips the only signal that would surface a broken prompt.
The Model-Preference Fork: Why Your Prompt Library Has Three Versions and No One Is Tracking the Drift
A shared prompt library quietly accretes model-specific forks that nobody tracks, breaking the contract between your eval suite and your routing layer at every model upgrade.
Model Rollback Velocity: The Seven-Hour Gap Between 'This Upgrade Is Wrong' and 'Old Model Fully Restored'
Rolling back an LLM upgrade isn't a button press — it's a partial, hysteretic operation closer to a database migration. Here's the control plane your incident playbook needs before the next bad model rolls out.
LLM Model Routing Is Market Segmentation Disguised As A Cost Optimization
Routing 60% of LLM traffic to a cheaper model bends the cost graph — and silently splits your AI feature into two products. The aggregate accuracy metric averages over the segment that gets hurt, two failure modes show up as one bug report, and customers experience two assistants with no release notes.
Multilingual Eval Cost Amplification: Why Seven Locales Doesn't Cost 7×
Your English eval suite cost $40K. The seven-locale international launch will not cost $280K — the real curve is closer to N×L^1.3 because cross-locale comparison is a meta-eval that doesn't decompose.
Your On-Call Rotation Needs an AI-Literacy Prerequisite Before It Pages Anyone at 2am
Shared on-call rotations break the moment one of those services is an LLM-backed feature. Here is the literacy prerequisite, dashboard hygiene, and shadow-period playbook that keeps the AI team out of bed at 2am.
On-Device AI Needs a Fleet Manager, Not a Model Card
Shipping one on-device model to every user means you're either burning battery on flagships or shipping a degraded product on the long tail. The discipline that fixes it looks more like a CDN than a model registry.
Pagination Is a Tool-Catalog Discipline: Why Agents Burn Context on List Returns
Tools that return unbounded lists turn agents into the SELECT * antipattern of the function-calling era. Pagination is a load-shedding primitive — make it a convention in your tool catalog, not a per-tool decision.
Per-Vector Version Tags: The Missing Column Behind Every Embedding Migration
Vector stores ship without the migration tooling Postgres has had for two decades — no ALTER TABLE, no online schema change, no per-row version. The discipline that makes embedding upgrades survivable starts with a single column most teams forget to add.
Prompt Cache Thrashing: When Your Largest Tenant's Launch Triples Everyone's Bill
Prompt caching's discount is real until one tenant's launch evicts everyone else's prefixes. The shared inference cache is a tenant-coupling surface, and the bill lands weeks after the incident.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
