Page 25
12 articlesBring-Your-Own-Key for AI Features: The Sales-Driven Re-Architecture Nobody Costed
BYOK looks like an auth toggle but moves your trust, cost, and operational boundaries at once. Here's the architecture work most teams underprice.
The Composability Tax: Why Adding Tools Makes Your Planner Worse
Every tool you add bends your planner's accuracy curve downward. The fix is a retirement metric — frequency × success × downstream lift — and a single owner of the catalog.
The Customer-Facing AI Postmortem When Nothing Crashed
Status page green, error rate zero, customers still unhappy. A field guide to writing AI quality-regression postmortems when nothing crashed — root-cause vocabulary, severity scales, and a follow-up cadence that closes the loop.
The Demo Account Eval Set Your Sales Team Is Running Without You
Sales demo accounts are a business-critical, unowned eval set — and they're how model migrations quietly break six-figure prospect walkthroughs. Here's the pattern to make them a first-class release gate.
When Marketing Reads Your Eval Cases: The Cross-Functional Visibility Problem
Your eval set is read by sales, marketing, legal, and customer success — and they each extract a different artifact than you intended. Build the engineering-vs-shareable split before a customer recognizes their own complaint in a procurement deck.
The First 90 Days for an AI Engineer: An Onboarding Playbook That Survives the Six-Week Doc Rot
A 90-day onboarding plan for AI engineers that swaps stale architecture docs for shadowed eval reviews, supervised prompt diffs, and end-to-end judge calibration.
Free-Tier Traffic Is Your Real Eval Set
Grading an LLM against paid-cohort traces is grading on the easy distribution. The cohort still deciding whether to upgrade lives in the free tier — and the eval set ignores it.
GPU Capacity Is a Roadmap Constraint: The 18-Month Contract That Decided Q3
Multi-year GPU commitments quietly bind product roadmaps to capacity decisions made by people who never saw the feature list. Here is the planning discipline that closes the gap.
The Internal Eval Set Is a Privacy Boundary Nobody Reviewed
AI teams routinely grade models against production conversations and call it an internal dataset. Under purpose limitation rules it is a separate processing event that nobody reviewed.
Latency-Aware Tool Selection: When 'Good Enough Now' Beats 'Best Available Later'
Static tool descriptions in agent prompts decay against live latency and error rates. A runtime cost-of-waiting signal in the prompt is what turns tool selection from a frozen eval artifact into a routing decision.
Locale-Stratified Evals: How to Catch Non-English Regressions Your English Test Set Can't See
An English-first eval rollup hides regressions on French, Japanese, and Portuguese queries until churn surfaces them. The discipline of locale-stratified evals, locale-conditioned judges, and traffic-weighted reporting catches locale drift before users do.
MCP Tool Deprecation: Why the Model Still Calls the Old Name
Renaming an MCP tool is not an API deprecation — it's a model-distribution shift. Why the old name keeps showing up and how to phase it out without paging on-call.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
