Your Model's Confidence Score Is a Vibe, Not a Probability

A support agent is about to issue a refund. Before it fires the tool call, your team added a gate: only proceed if the model says it's at least 90% confident. The model dutifully returns "confidence: 0.95," the refund goes out, and the citation it used to justify the amount — a policy clause about damaged goods — does not exist. It never did. The model invented the clause and then rated its own invention at 95%.

This is the trap. Teams reach for the model's confidence number because it looks like the probability you'd get from a calibrated classifier — a value where 0.9 means "right nine times out of ten." It is not that. An LLM's self-reported confidence is a fluent-sounding token like any other, shaped by tone, phrasing, and training incentives that have almost nothing to do with whether the underlying claim is true.

If you are gating real actions on that number, you are gating on a vibe.

The confusion is understandable. In classical machine learning, a model's output probability is a real quantity you can hold accountable: bin every prediction where the model said "0.8," check how many were correct, and a well-behaved model lands near 80%. That property is called calibration, and it is the thing you actually want when you write if confidence > threshold. The problem is that nobody on your team measured whether your LLM has it, and the default answer is that it does not.

Calibration Is the Property You Never Measured

Calibration asks one question: when the model says P, is it right P fraction of the time? You test it with a reliability diagram. Bucket every prediction by its stated confidence, plot the average stated confidence on the x-axis against the observed accuracy on the y-axis, and a perfectly calibrated system traces the 45-degree diagonal. Points below the line mean overconfidence — the model claims more certainty than it earns.

The single-number summary of that gap is Expected Calibration Error (ECE): the average absolute distance between confidence and accuracy across the buckets. An ECE near zero is good. An ECE of 0.2 means your confidence numbers are off by twenty points on average, which makes a "0.9" gate functionally meaningless.

Here is what the research finds when people actually run this test on production-grade models. RLHF-tuned models overwhelmingly emit verbalized confidence between 80% and 100% — they have essentially forgotten how to say "I'm not sure" — with ECE reaching 0.30 or higher on knowledge-intensive tasks. Even the better-behaved models are imperfect: a recent medical question-answering study measured mean ECE ranging from about 0.06 for the strongest model to 0.13 for a popular alternative, and that is on a benchmark, under controlled conditions, with the model's introspection working as well as it ever does.

The uncomfortable takeaway: the model is often least accurate exactly when it sounds most sure. Researchers have named this the Dunning-Kruger effect of language models, and it is not a bug you can prompt away. It is baked in by how the models are trained.

Why RLHF Manufactures Confidence

The overconfidence is not random noise; it is a trained-in incentive. During reinforcement learning from human feedback, a reward model scores candidate responses, and reward models exhibit a systematic bias toward answers that sound confident regardless of whether they are correct. A hedge-free, assertive answer gets a higher reward than an accurate-but-tentative one. Optimize against that signal for long enough and you get a policy that has learned to project certainty as a style, decoupled from the truth of the content.

This is why the confidence number moves with things that have nothing to do with correctness. Rephrase the question and the number shifts. Change the temperature and it shifts. Add "you are an expert" to the system prompt and it shifts. Confidence elicitation has even been studied as an attack surface — you can manipulate the stated confidence through prompt engineering alone, which is only possible because the number was never anchored to anything real.

Loading…