2 posts tagged with "instrumentation"

The first person to notice your reasoning-model regression is almost never on the engineering team. It is the finance analyst who pings your manager on a Tuesday afternoon because the previous month's Anthropic invoice came in 2.4x higher than the prior one, and "we didn't ship anything that should have done that." You open the dashboard, look at request volume — flat. Latency p99 — flat. Output tokens per response — flat. Error rate — flat. Every panel you wired up six months ago says the system is healthy. Finance is looking at a different number, and they are right.

The number they are looking at is reasoning tokens, and most observability stacks were built before the field existed.

Your AI feature launched three months ago. DAU is up. Session length is climbing. Your dashboard looks green. But here is the uncomfortable question: are your users actually adopting the feature, or are they just tolerating it?

Most teams track AI feature adoption with the same metrics they use for traditional product features — daily active users, session duration, feature activation rates. These metrics worked fine when features behaved deterministically. Click a button, get a result, measure engagement. But AI features are fundamentally different: their outputs vary, their value is probabilistic, and users develop trust (or distrust) through repeated exposure. The standard metrics don't just fail to capture this — they actively mislead.