Page 30
12 articlesSnapshot Eval Decay: When Green CI Stops Meaning Your Product Still Works
A green eval suite that ran for six months may already be testing yesterday's product against yesterday's reality — here is how snapshot eval decay hides in plain sight and how to keep an eval set alive.
The Streamed-Response Trace Schema Gap: Why Your APM Lies About LLM Latency
Streaming LLM responses break the request/response span model. The duration field lies; failures live between the boundaries — TTFT regressions, mid-stream stalls, content loops — and the fix is checkpointed token-time events with a real tail-event taxonomy.
Tenancy Leaks Through Few-Shot Examples: When Your Prompt Library Becomes a Cross-Customer Data Store
Mining production traces for few-shot examples quietly turns your system prompt into an unaudited multi-tenant data store. Here is how the leak happens, why it is a contract breach, and the discipline that catches it before a customer does.
The Agentic Stamp: When Marketing Names It and Engineering Pays the Operational Bill
Marketing calls a workflow an agent, and engineering inherits the observability, tool-budget, and escalation work nobody scoped — a leadership decision dressed up as a naming choice.
Token Accounting Drift: When Your Trace Logs Don't Match the Provider Invoice
Every team building on a hosted LLM eventually finds the token counts in their traces don't match the monthly invoice. The gap is rarely fraud — it's a structural measurement problem with six compounding causes.
Two-Hop Tool Chains: Why 95% Tools Compose Into 80% Pipelines
Per-tool dashboards stay green while end-to-end agent reliability collapses. The failure lives at the seams between tools, where contract drift, pagination, and unit mismatches turn 95% primitives into 80% pipelines.
User Trust Half-Life: Why One Bad Session Erases Weeks of Calibration
AI users spend weeks building trust calibration and lose it in one bad session. Build telemetry for verification, undo, and engagement-without-action to catch trust erosion before churn shows up.
The Vendor SLA Gap: Why Your LLM Provider's Uptime Misses the Failure Mode That Breaks Your Product
Your LLM vendor's 99.95% uptime number does not cover refusal spikes, silent model bumps, or quota-driven degradation. Here is the functional-availability instrumentation that does.
Your Fine-Tuning Corpus Is a Codebase. Stop Shipping It Through a Bucket.
Application code has PR review, signed commits, and named authors. Fine-tuning corpora have an S3 bucket and a Mechanical Turk batch from 2024. The threat models are inverted, and 250 documents can backdoor a 13B model.
Escalation Rate Is the Eval Signal Your Offline Tests Missed
Escalation rate is one of the few honest signals of agent capability, but in most companies it lives on the ops team's staffing dashboard — not the AI team's eval review. Here's how to close that gap.
On Intelligence, Chapter by Chapter: A 2004 Book That Predicted Half of Modern AI
A chapter-by-chapter walk through Jeff Hawkins' On Intelligence — what it got right about prediction-as-cognition, what it missed about scaling, and why it still frames how I read transformer behavior in 2026.
The Platform-Readiness Gap: When AI Features Ship Before the Infra to Operate Them
AI features ship before the platform exists to operate them, and the debts compound. A launch gate, named owners, and a deliberate platform ramp are the only way out.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
