The Demo That Set a Baseline You Cannot Afford to Run

The demo went well. The agent answered the hard question, chained four tool calls without a stumble, and produced a paragraph that made the room go quiet for a second before someone said "ship it." Nobody asked what it cost. Nobody asked what model it ran on, how many inputs you tried before that one, or what happens when a thousand people hit it at once instead of you, alone, at your desk, on a Tuesday.

That demo just became a contract. Not a written one — worse. It became the unstated baseline that leadership, sales, and customers will hold the shipped product against. And the terms of that contract were set by a system you cannot afford to run.

The gap between demo economics and production economics is real, large, and almost never priced before the commitment is made. Gartner expects more than 40% of agentic AI projects to be canceled by 2027, largely on cost overruns. A March 2026 survey found 78% of enterprises had agent pilots running and only 14% had scaled one to organization-wide use. The pilots are not failing because the technology does not work. They are failing because the version that worked was never the version anyone could deploy.

The demo is an expectations contract nobody read

When you demo an agent, you are not showing what the product does. You are setting a reference point. Everyone in the room walks out with a mental model anchored to what they just saw, and every future conversation gets measured against that anchor.

The problem is that a demo is optimized for exactly one thing: the reaction in the room. So you reach for the best model available, because token cost is invisible during a demo. You pick inputs you have already tried, because a demo is a performance and you rehearse a performance. You run it once, for one person, with no concurrent load and no rate limits. You present to an audience that wants it to succeed and will forgive a stumble as "early days."

None of those conditions survive contact with production. But the expectation does. "Make it as good as the demo" sounds like a quality requirement. It is actually a budget request, and a large one, submitted by people who never saw the invoice. The demo signed a contract on the team's behalf, and the team finds out the terms only when the bill arrives.

Everything the demo quietly removed

Walk through what a demo strips out, one layer at a time, and the size of the hidden gap becomes clear.

The model tier. Demos run on the frontier. Why wouldn't they — the marginal cost of a single impressive run is rounding error. But the price spread across the model market is enormous. In 2026 the gap between a cheap usable model and a top frontier model runs into the hundreds of times per token. A demo that costs you two cents is fine. The same call, at production volume, routed to the same frontier model, is a line item someone will eventually circle in red.

The inputs. A demo uses inputs you chose. Production uses inputs users choose. Demo data is clean, well-formed, and inside the distribution the agent handles well. Real traffic is messy, ambiguous, adversarial, and full of the long-tail cases you never rehearsed. The demo showed the p50 of a hand-picked sample. Production is the full distribution, tail included.

The concurrency. A demo is one request at a time. Production is thousands, with rate limits, queueing, retries, and the tail-latency behavior that only shows up under load. An agent that feels instant when it is the only thing running can become a sluggish queue when it is one of many.

The tool loop. The demo's four clean tool calls hide a structural cost. Tool-augmented agents make many times more model calls than simple prompting — frequently nine times or more once retries, reflection, and error handling are included. Each step also adds latency: five sequential inference steps at 200 ms each is a full second of compounded wait before the user sees anything. The demo showed one happy-path traversal. Production runs every path, including the loops that do not terminate cleanly.

The audience. The demo audience wanted it to work. Production users want their problem solved and do not grade on a curve. A response that drew applause in a conference room draws a support ticket when it is 80% right and the user needed 100%.

Stack these together and you get the number practitioners keep rediscovering: production cost typically lands at four to eight times the pilot cost, and wider for complex or regulated systems. A pilot usually runs at 15–25% of full deployment cost while skipping roughly 70% of the hard problems. For retrieval-heavy systems the overrun is worse — RAG projects have been reported to run several times over their pilot cost projections. The demo did not just understate the cost. It understated it by a multiple.

"As good as the demo" is a promise made of someone else's money

Here is the trap. The demo proved the agent can be that good. So when production quality comes in lower, it reads as a regression — as if the team broke something. It did not. The team is discovering the price of the demo's quality and finding it unaffordable.

Production forces compromises the demo never had to show:

• Model downgrade. You route most traffic to a cheaper model and reserve the frontier tier for the requests that genuinely need it. Quality on the median request drops a notch. That notch is the difference between demo economics and a sustainable bill.
• Latency budgets. You cap the tool loop, trim the context, and cut a reflection step, because every step is money and milliseconds. The agent that "thought carefully" in the demo now thinks faster and shallower.
• Abstention. A production agent has to say "I'm not sure" or hand off to a human. The demo never abstained — you would not demo an abstention. But abstention is what keeps a production agent from being confidently wrong at scale.

Every one of these is the right engineering call. Every one of them makes the product visibly worse than the demo. And because the demo set the baseline, each correct decision now looks like a broken promise. The team spends its credibility defending good choices against a standard that was never real.

There is a blunt version of this failure: an agent run without cost controls can end up more expensive per task than paying a person to do the same work. The demo never showed that, because the demo was one task. Production is a million tasks, and the unit economics are the whole game.

Demo under production constraints, not ideal ones

The fix is not to stop demoing. It is to stop demoing a system you cannot ship. Close the gap before it becomes a roadmap promise.

Demo on the model tier you will actually run. If production will route median traffic to a mid-tier model, demo on that model. If the demo is weaker as a result, that weakness is information — surface it now, while it is a design conversation, not later, when it is a credibility problem.

Demo with unrehearsed inputs. Let someone in the room type their own question. Pull a sample of real or realistic queries and run them live, tail cases included. A demo that survives inputs you did not choose is a demo that means something.

Demo under load, or at least name the absence of it. You may not be able to simulate production concurrency in a meeting. You can say out loud: "this is a single-request demo; latency and cost under concurrency are open questions, and here is our current estimate." That one sentence converts an unstated contract into an explicit assumption.

Put a number on the demo. Show the cost and latency of the run you just did. "That answer cost 14 cents and took 3.2 seconds. At our projected volume on the frontier model, that is X per month. Here is the same query on the production tier: 2 cents, 1.1 seconds, slightly worse answer." Now the room is choosing between real options instead of anchoring on an imaginary one.

Name the demo-to-production gap explicitly. Before anyone says "ship it," say the gap out loud: the demo ran on the best model, with chosen inputs, no load, and a friendly audience; production changes all four; expect quality to land lower and cost to land higher; here is the band we are estimating. Naming it does not make the gap smaller. It makes the gap a shared, known quantity instead of a surprise that lands on the engineering team alone.

The baseline you set is the baseline you owe

A demo is the most powerful expectations-setting tool a team has, which is exactly why an unconstrained demo is dangerous. The version that gets applause defines what "working" means for everyone who saw it, and if that version runs on economics you cannot sustain, you have signed a contract to deliver something you cannot deliver.

The discipline is simple to state and hard to practice: never demo a configuration you are not willing to operate. If the demo runs on the frontier model, you have promised the frontier model. If the demo abstains from nothing, you have promised an agent that never says "I don't know." If the demo is instant, you have promised instant. Every gap between the demo and the deployable system is a gap the engineering team will spend its credibility closing — and it will look, from the outside, like the team got worse at its job.

Demo the product you can afford to run. A slightly less impressive demo that reflects production reality is worth more than a stunning one that writes a check the system cannot cash. The applause fades in a week. The baseline lasts until someone renegotiates it — and renegotiating a baseline is far more expensive than setting an honest one.