Blog
Insights, analysis, and updates from the AI agent economy. Browse by tag.
The Demo Account Eval Set Your Sales Team Is Running Without You
Sales demo accounts are a business-critical, unowned eval set — and they're how model migrations quietly break six-figure prospect walkthroughs. Here's the pattern to make them a first-class release gate.
When Marketing Reads Your Eval Cases: The Cross-Functional Visibility Problem
Your eval set is read by sales, marketing, legal, and customer success — and they each extract a different artifact than you intended. Build the engineering-vs-shareable split before a customer recognizes their own complaint in a procurement deck.
The First 90 Days for an AI Engineer: An Onboarding Playbook That Survives the Six-Week Doc Rot
A 90-day onboarding plan for AI engineers that swaps stale architecture docs for shadowed eval reviews, supervised prompt diffs, and end-to-end judge calibration.
Free-Tier Traffic Is Your Real Eval Set
Grading an LLM against paid-cohort traces is grading on the easy distribution. The cohort still deciding whether to upgrade lives in the free tier — and the eval set ignores it.
GPU Capacity Is a Roadmap Constraint: The 18-Month Contract That Decided Q3
Multi-year GPU commitments quietly bind product roadmaps to capacity decisions made by people who never saw the feature list. Here is the planning discipline that closes the gap.
The Internal Eval Set Is a Privacy Boundary Nobody Reviewed
AI teams routinely grade models against production conversations and call it an internal dataset. Under purpose limitation rules it is a separate processing event that nobody reviewed.
Latency-Aware Tool Selection: When 'Good Enough Now' Beats 'Best Available Later'
Static tool descriptions in agent prompts decay against live latency and error rates. A runtime cost-of-waiting signal in the prompt is what turns tool selection from a frozen eval artifact into a routing decision.
Locale-Stratified Evals: How to Catch Non-English Regressions Your English Test Set Can't See
An English-first eval rollup hides regressions on French, Japanese, and Portuguese queries until churn surfaces them. The discipline of locale-stratified evals, locale-conditioned judges, and traffic-weighted reporting catches locale drift before users do.
MCP Tool Deprecation: Why the Model Still Calls the Old Name
Renaming an MCP tool is not an API deprecation — it's a model-distribution shift. Why the old name keeps showing up and how to phase it out without paging on-call.
Mobile App Store Review Meets AI Features: The Deploy Cadence Collision
Web AI features iterate in minutes; mobile AI features iterate on the platform's review clock. The architectural seams that keep both surfaces honest under one eval set and two release trains.
Model Migration Bills You Twice: The Eval Re-Anchoring Tax Nobody Prices
Swapping foundation models silently invalidates your eval baseline — human-anchored scores, LLM judges, snapshots, and team intuition all need re-anchoring, and the labor bill is usually larger than the per-token savings.
On-Call at 3am for an AI Feature That Didn't 500
The pager fires because eval-on-traffic dropped four points, not because a service crashed. Runbook patterns, alert design, and rotation discipline for the failure modes that don't trip your existing alerts.