Forrester says AI hype is fading as vendor promises don't match delivered value. Shocking... or long overdue market correction?

General
1

As CTO at a mid-stage SaaS company, I’ve sat through probably 100+ AI vendor pitches in the past 18 months. I’ve also deployed about a dozen AI tools across our engineering and product organizations.

The gap between vendor demos and production reality has been… educational.

The Promise vs Reality Pattern

Here are three examples from our actual vendor evaluations:

AI Code Review Tool

  • Vendor promise: “Catch 90% of bugs before code review, reduce review time by 60%”
  • Reality after 6-month pilot: Caught 35% of the bugs our senior engineers would catch, with a 15% false positive rate that created noise
  • Actual impact: Modest time savings (around 20%), useful but nowhere near transformative

Customer Support AI

  • Vendor promise: “Resolve 70% of support tickets automatically, 10x productivity per agent”
  • Reality after deployment: Handled 40% of tickets (mostly simple password resets and FAQ lookups we could have solved with better docs)
  • Actual impact: Freed up human agents for complex issues, but definitely not 10x anything

AI-Powered Analytics Platform

  • Vendor promise: “Business users can query data in natural language, no SQL needed”
  • Reality: Required significant prompt engineering and domain knowledge to get useful results. Complex queries still needed data analysts.
  • Actual impact: Slight democratization of simple metrics, but not the transformative “no-code analytics” promised

Why This Matters For CFO Skepticism

Here’s the problem: Vendor over-promising creates unrealistic expectations across the organization.

Our CEO sees a demo showing “10x productivity.” Our board hears competitors claiming transformative AI capabilities. Our CFO is told AI will dramatically reduce costs.

Then we deploy the tools and see… 15-25% improvements. Real value! But nowhere close to the promised transformation.

So when I ask for AI budget and promise measurable impact, the CFO is understandably skeptical. We’ve been burned before by the gap between promise and reality.

Forrester’s prediction that AI hype is fading is actually a healthy market correction. It’s forcing vendors to make realistic claims and buyers to set realistic expectations.

What Changed in How We Evaluate Vendors

After learning these lessons the expensive way, here’s our updated approach:

1. Demand Customer References With Actual Metrics
Not testimonials. Not case studies written by vendor marketing. Actual customers willing to share:

  • What they measured before deployment
  • What they measured after
  • What it actually cost (including integration and operational overhead)
  • What problems they encountered

If vendors can’t produce 3+ references willing to have detailed conversations, that’s a red flag.

2. Pilot Programs With Clear Success Criteria
We now structure pilots as experiments, not trials:

  • Define success metrics upfront (before seeing vendor solution)
  • Establish baseline measurements
  • Run controlled pilots (portion of team/users/workflow with vendor tool, portion without)
  • Measure incrementally: what lift did the AI actually provide?
  • Set kill criteria: if we don’t see X improvement in Y timeframe, we walk away

3. Account for Total Cost of Ownership
Vendor pricing is rarely the actual cost. We now model:

  • Integration engineering time (usually 2-4 weeks minimum)
  • Training and change management
  • Ongoing maintenance and support
  • Infrastructure costs for deployment
  • Opportunity cost of engineering time vs other priorities

That $50K/year tool often becomes $150K all-in when you account for everything.

4. Benchmark Claims Independently
We’ve started using independent benchmarks and academic research to validate vendor claims before pilots.

If a vendor claims “90% accuracy” on their marketing site but independent benchmarks show 60% in similar use cases, we negotiate from a position of reality.

My Question For The Community

What’s your BS detector for AI vendor claims? What questions do you ask? What red flags make you walk away?

And for vendors reading this: Please help us by setting realistic expectations. Under-promise and over-deliver builds trust. The opposite creates the skepticism we’re seeing from CFOs across the industry.

2

Michelle, from the finance side, I want to add a dimension to this vendor relationship conversation: We’re now demanding performance guarantees in contracts.

The Shift in How Finance Negotiates AI Deals

Six months ago, we evaluated AI vendors based on their pitch and price. Standard enterprise software procurement.

Now, our CFO requires:

  • Usage-based pricing whenever possible (pay for outcomes, not seats)
  • Performance guarantees with financial remedies if metrics aren’t hit
  • Pilot-to-production gates where we can walk away before committing to annual contracts

A Concrete Example

We recently evaluated a sales intelligence AI tool. Initial pitch: “20% increase in qualified leads, 30% improvement in conversion rates.”

Our contract terms:

  • 3-month pilot with our success metrics defined upfront (not theirs)
  • Payment milestone 1 (50%): After pilot shows ≥15% improvement in lead qualification accuracy vs our baseline
  • Payment milestone 2 (50%): After 6 months production usage sustaining ≥15% improvement
  • Exit clause: If metrics degrade below baseline for 2 consecutive months, we terminate with pro-rated refund

The vendor initially pushed back. “Our standard contract doesn’t include performance guarantees.”

Our response: “Then you’re not confident enough in your own product claims for us to bet budget on it.”

They eventually agreed to modified terms. And honestly? It made them more invested in our success. They assigned a dedicated success engineer to ensure we hit the metrics.

Red Flags From Finance Perspective

What makes me walk away from vendor conversations:

1. Unwilling to Share Customer Success Data
If you can’t show me 5+ customers who achieved the outcomes you’re promising, with data, you’re not mature enough for our enterprise bet.

2. Pricing That Doesn’t Scale With Value
Seat-based pricing for productivity tools is fine. But for tools promising business outcomes (revenue, conversion, retention), I want value-based pricing.

3. No Risk-Sharing
If the vendor won’t put their fees at risk based on delivering value, why should we put our budget at risk?

Michelle, your point about total cost of ownership is critical. From finance perspective, I now require engineering to provide fully-loaded cost estimates including integration time before we even start vendor evaluation.

The $50K tool that becomes $150K all-in won’t get approved unless the business case justifies $150K.

3

Michelle, as someone who’s evaluated dozens of AI coding assistants, ML platforms, and analytics tools for our product data science work, I want to add the technical evaluation dimension.

Vendor Claims vs Independent Benchmarks

The most important thing we’ve learned: Don’t trust vendor benchmarks. Run your own.

Vendors optimize their benchmarks for marketing. They choose:

  • Test datasets where their model performs best
  • Evaluation metrics that favor their approach
  • Comparison points against weak baselines

We now have a standard protocol:

1. Evaluate on OUR Data, Not Theirs
We create representative samples from our actual production data (properly anonymized).

An AI coding assistant might perform great on public GitHub repos but struggle with our internal codebase style and domain-specific patterns.

2. Test on Edge Cases and Failure Modes
Vendors demo the happy path. We specifically test:

  • Ambiguous queries where there’s no clear right answer
  • Complex domains requiring specialized knowledge
  • Edge cases and unusual inputs

This is where accuracy claims of “90%” often collapse to 40-50%.

3. Compare Against Simple Baselines
Sometimes a vendor’s AI solution is barely better than:

  • Rule-based systems
  • Simple heuristics
  • Basic statistical approaches

We built a “dumb” baseline for customer support ticket routing using keywords and routing rules. Took one engineer two weeks. The AI vendor’s solution was only 8% more accurate, at 50x the cost.

We kept our simple system.

Warning Signs in Technical Evaluation

Red flags that make me skeptical:

1. “Proprietary AI Magic”
If they can’t explain how their system works at a technical level, I don’t trust it. Black box AI in production is a liability.

2. Claiming Near-Perfect Accuracy
Nothing is 90%+ accurate in real-world messy data. Claims like this suggest they haven’t deployed in production at scale.

3. No Discussion of Failure Modes
Every AI system fails. If the vendor can’t articulate when and how their system fails, they either don’t know (bad) or won’t tell you (worse).

4. Benchmark Gaming
Watch for suspiciously specific benchmark setups: “92.3% accuracy on ImageNet subset filtered for high-confidence classifications” = they tuned until they got a number they liked.

What I Actually Want From Vendors

Be honest about:

  • What your system is good at AND what it struggles with
  • Real-world accuracy ranges (not best-case)
  • Computational and latency costs at scale
  • How much domain-specific fine-tuning will be needed

Under-promise and over-deliver. The vendors who do this are the ones we build long-term relationships with.