The Eval That Scored Every Turn and Missed the Whole Conversation

Your eval dashboard is green. Turn-level accuracy sits at 95%, the LLM judge agrees with your annotators, and every regression test passes on the way to production. Then a user files a bug: the agent recommended a Postgres index on turn nine that directly contradicted the "we're on DynamoDB" constraint the user set on turn one. You pull the transcript. Every single turn, read in isolation, is a reasonable response. The conversation, read as a whole, is a disaster.

This is the central lie of turn-level evaluation. It grades request-response pairs because that's the unit that's cheap to label, and it quietly assumes that a conversation is just a bag of independent turns you can average. It isn't. The response at turn nine is conditioned on everything that came before it, and the failures that actually reach users almost never live inside a single turn — they live in the seams between turns, where state gets dropped, assumptions harden, and small errors compound into a wrong final answer.

If you only remember one thing: a system that is 95% correct per turn is not 95% correct per session. It's much worse, and the gap widens with every exchange. Most eval harnesses are structurally blind to exactly the failure mode that defines whether a multi-turn agent is usable.

The Arithmetic That Turns 95% Into 77%

Start with the compounding. If each turn succeeds independently with probability 0.95, a five-turn session succeeds with probability 0.95⁵ ≈ 0.77. Push per-turn accuracy to 99% and the five-turn number climbs back to about 0.95. That single percentage point of per-turn reliability is the difference between "usually works" and "usually fails" once you chain enough turns together.

And independence is the optimistic assumption. Real conversations are worse than the multiplication suggests, because errors don't just accumulate — they propagate. A 2% misalignment introduced at turn two doesn't stay a 2% problem. If the model builds on the wrong assumption instead of reconsidering it, that seed can bloom into a 40% failure rate by turn ten. Practitioners auditing production transcripts consistently find that 70–90% of multi-turn errors trace directly back to a previous response, not the current one. The model latches onto something early — "I'm working on a Python project," mentioned once in turn one — and keeps applying it long after the user has moved to a different file, a different service, a different question entirely.

Turn-level evals cannot see any of this. Each turn gets graded against its immediate prompt, the wrong turn nine looks locally fine given the poisoned context it inherited, and the score sails through green. You have measured local correctness and called it quality. They are not the same thing.

Why Averaging Turn Scores Is Statistically Broken

There's a subtler problem, and it's not just a coverage gap — it's a math error. The dominant paradigm computes a score per turn and averages across the conversation. That averaging silently assumes the turns are independent samples. They are the opposite of independent: turn t is conditioned on the full history up to t, and the user's next message is shaped by what the model just said.

When you run standard statistical analysis over turn-level scores as if each turn were an independent observation, autocorrelation wrecks your conclusions. One recent analysis estimated that 42% of turn-level findings in conversation analysis may be spurious — artifacts of treating dependent turns as independent draws. You are not just missing failures. You are generating confident, wrong conclusions about which prompt change helped and which hurt, then shipping on them.

Aggregation also hides catastrophe by construction. A session that goes perfectly for nine turns and then commits a fatal error on turn ten scores 90% under averaging. But that conversation is a total loss — the user got a wrong answer and left. Averaging rewards local fluency and buries the one turn that actually determined the outcome. The metric that looks the most reassuring is the one most likely to be lying to you.

What Trajectory-Level Evaluation Actually Measures

Loading…