Unpopular Opinion: Maybe We Are Obsessing Over AI ROI Instead of Building Products

maya_builds · March 17, 2026, 7:34am

I am going to say something controversial. What if we are spending more time measuring AI productivity than actually using AI to build better products?

At my last startup, we spent three months building an AI productivity dashboard. We tracked everything. But you know what we did not do? Ship the features our customers were asking for.

Our product velocity actually slowed down because we were so busy measuring productivity that we forgot to be productive.

The Designer Perspective

I am a designer and the best design tools I have used resist quantification. When Figma introduced auto-layout did anyone measure the ROI? No we just started using it because it made our work better.

AI tools like Claude Cursor and v0 have changed how I prototype. But if you asked me to quantify the value I honestly could not tell you. And I am not sure it matters.

Some of the best work happens in unmeasurable spaces. How do you measure when AI helped you think differently about a problem?

Acknowledging Reality

I get it. CFOs need numbers. Boards need ROI. Michelle Luis and Keisha are right that leaders need to speak CFO language.

But I am worried about the distraction cost.

The Middle Ground

Maybe we need lightweight tracking instead of comprehensive frameworks.

Just track adoption satisfaction quality and a few key business metrics. Then trust our engineers to use the tools that make them more effective.

The Provocation

Are we building AI measurement systems instead of AI-enhanced products? Are we having more meetings about AI ROI than meetings about customer problems?

At some point does the measurement apparatus become more expensive than the thing we are measuring?

The Question

How do you measure enough to justify investments without letting measurement become the work itself?

When do we trust that good tools in good hands will create value even if we cannot measure every dimension?

Unpopular opinion I know. But someone had to say it.

product_david · March 17, 2026, 7:35am

Maya I appreciate you bringing this perspective. It is a necessary counterbalance to the measure everything mindset that can spiral into measurement theater.

But I am going to respectfully push back while also finding the middle ground.

Where I Disagree

Without metrics investments get cut arbitrarily.

Let me share what happened to us last year. We had GitHub Copilot subscriptions for our whole engineering team—$240K annual spend. Usage was high. Developers loved it. Anecdotal feedback was overwhelmingly positive.

Then our Series B funding round fell through. We had to cut $2M from our annual budget. Every department had to justify their discretionary spending.

The design team kept their Figma licenses because they could show we design 40 percent more variations per project and A/B testing shows these variations improved conversion by 12 percent which is X dollars in ARR.

We lost Copilot because all we could say was developers like it and feel more productive. That was not enough when finance was making hard choices.

Three senior engineers left within six months. Two cited lack of modern tooling in their exit interviews. Replacing them cost us over $400K in recruiting and onboarding plus 6 plus months of productivity loss.

The lack of metrics cost us more than the tool subscription.

Where I Agree

Measurement can become performative if it is not tied to decisions.

Your startup example is a cautionary tale. Building dashboards that nobody uses to make decisions is pure waste. Measuring for measurement sake is theater.

The key question: What decision does this metric inform?

If the answer is none do not measure it. If the answer is whether to renew this tool subscription then measure the minimum needed to make that decision confidently.

The Middle Ground

I think you are exactly right about lightweight but rigorous measurement.

Here is what I would propose as the minimum viable AI measurement:

Quarterly Review Questions:

Are people using the tool? (Adoption rate)
Do they want to keep it? (Satisfaction survey)
Are outcomes improving? (Pick 2-3 key metrics)
What is the business impact? (Rough estimate not precise calculation)

That is it. Four questions. One hour per quarter. No daily tracking no comprehensive dashboards no productivity surveillance.

But those four questions answered honestly give you enough to defend the investment when budget cuts come.

When Measurement Becomes Too Much

You asked when measurement becomes more expensive than value. Here is my heuristic:

If you are spending more than 5 percent of the value gained on measuring the value you are over-measuring.

For a $200K tool investment generating roughly $800K in value you should spend no more than $40K of effort measuring and reporting. That is maybe 200-300 hours per year across the whole organization.

If your measurement overhead exceeds that cut back to quarterly reviews and lightweight tracking.

On Creativity and Exploration

Your point about unmeasurable creative value is real and important.

But here is the synthesis: measure the foundational tools trust the exploratory tools.

GitHub Copilot (core productivity tool) measure rigorously. Claude for brainstorming (exploratory tool) light tracking trust the team. Cursor for production code (core tool) measure outcomes. v0 for rapid prototyping (exploratory) light touch.

Different tools different measurement approaches.

The Real Question

You asked when do we trust that good tools in the hands of good people will create value?

My answer: Always. But also verify.

Trust does not mean no measurement. It means smart measurement that informs decisions without creating overhead burden.

Your contrarian take is valuable Maya. We do need to be careful that measurement does not become the work. But the answer is not no measurement—it is strategic minimum measurement that protects the investments our teams need.

eng_director_luis · March 17, 2026, 7:36am

Maya I see both sides of this so clearly because I have lived both extremes.

The Over-Measurement Hell

Two years ago I worked at a company that tried to measure everything. We had daily AI usage reports emailed to managers, weekly productivity scorecards comparing developers, monthly AI ROI reviews that took 4 hours of prep, and quarterly board presentations requiring 40 plus hours of data analysis.

My team spent approximately 15-20 percent of their time feeding the measurement machine. Filling out surveys. Categorizing their work. Justifying why their AI productivity score was lower than another team.

Morale tanked. Developers felt micromanaged. The best engineers updated their LinkedIn profiles.

And you know what? The measurements were mostly noise anyway. We were measuring activity not outcomes.

So I get your frustration. I really do.

The Under-Measurement Disaster

But I have also seen the other extreme.

At a different company we had zero measurement for our tool investments. We operated on vibes and trust. If engineers wanted a tool we got it. If they said it was useful we believed them.

Then we had a bad quarter. Revenue missed projections. The exec team needed to cut 20 percent from department budgets.

Every tool subscription that could not be defended with data got cut. We lost Datadog we lost GitHub Advanced Security we lost our AI coding assistants. All in one brutal budget review.

Why? Because other teams could prove ROI and we could not.

The worst part: we probably had the data to prove value. But we had never bothered to track it systematically. When the CFO asked what is the impact of these tools all we had were anecdotes.

The Balance I Have Found

What is working for me now is what David called strategic minimum measurement.

What I track: Adoption rate (quarterly check-in 30 minutes), team satisfaction (quarterly survey 3 questions), 3 key outcome metrics (cycle time defect rate incident frequency—already tracked for other reasons), and rough business impact calculation (quarterly 2 hours of work).

What I do not track: Individual developer productivity scores, daily usage patterns, time saved per task, or granular feature-by-feature ROI.

The quarterly rhythm is key. It is frequent enough to catch problems early but infrequent enough that it does not become a distraction.

The Team Culture Component

Here is something that shifted for me: I reframed the measurement conversation with my team.

Instead of we need to justify these tools to finance I said these tools make your lives better. Let us collect the minimum data we need to keep them funded when budget cuts come.

That reframing changed everything. My team went from resisting measurement to actively helping with it because they understood it was protecting something they valued.

The Question You Asked

How do we measure enough without measuring too much?

I do not have a perfect answer but here is my heuristic:

If measurement feels like work you are measuring too much. If you cannot defend your budget when cuts come you are measuring too little.

The sweet spot is somewhere between those two extremes.

The Trust vs Verify Reality

You asked when we trust that good tools create value even if we cannot measure it.

In an ideal world? Always. In the world where CFOs make budget decisions? We need just enough measurement to protect the things our teams value.

I think of it like this: measurement is not surveillance. It is insurance. We are not measuring productivity to micromanage people. We are collecting evidence to defend the tools they need when budget pressure comes.

Question for You

Maya given your startup experience where measurement became a distraction what is the minimum measurement you think would have been useful?

If you could go back and design a lightweight system that protected your AI tools without becoming theater what would it look like?

I think there is wisdom in the creative exploratory space you describe. Maybe the framework is: measure the inputs and outputs but trust the creative process in between?

cto_michelle · March 17, 2026, 7:36am

Maya I appreciate the provocation. You are pushing back on something that absolutely can become dysfunctional if we are not careful.

Let me share the executive reality and then I will acknowledge where you are absolutely right.

The Fiduciary Responsibility Reality

As a CTO I have fiduciary responsibility to our board investors and ultimately our customers to spend money wisely. That is not optional. It is literally part of my job.

When we spend $400K annually on AI tools I am accountable for that investment. Not to micromanage how engineers use the tools but to ensure we are creating more value than we are spending.

The alternative—spending hundreds of thousands of dollars on tools with zero measurement—that is not trust. That is negligence.

But here is the crucial nuance: measurement should inform investment decisions not dictate daily work.

When I Over-Measured And Learned Better

Early in my CTO tenure I made exactly the mistake you are describing. We built an elaborate AI productivity dashboard. We tracked time saved per developer per day AI-assisted commits versus manual commits code quality scores weekly adoption reports and monthly ROI projections.

It was comprehensive. It was data-driven. It was also completely counterproductive.

Developers felt surveilled. Managers spent hours preparing reports. We had dashboard fatigue within six weeks. And worst of all it was not actually helping us make better decisions—it was just creating noise.

So I killed it.

The System That Works

What we do now: Three Core Metrics Quarterly Review.

Are teams using the tools? (Adoption rate)
Are key outcomes improving? (Cycle time quality incidents)
What is the business impact? (Revenue growth cost avoidance retention)

That is it. No daily tracking. No productivity surveillance. No comparative scorecards between developers.

Quarterly Deep Dive When Red Flags Appear: If adoption is low investigate why. If outcomes are not improving investigate friction points. If business impact is unclear dig into specific use cases.

But we only deep dive when there is a signal that warrants investigation. We do not deep dive as a matter of routine.

The Strategic Approach

Your point about creativity and exploration is important. Some of the best work happens in unmeasurable spaces.

Here is how I think about it:

Foundation tools (GitHub Copilot core infrastructure) measure consistently. Exploration tools (prototyping research learning) light touch trust the process. Experimental tools (trying new AI capabilities) measure adoption keep if teams want it.

Different purposes different measurement approaches.

Where You Are Absolutely Right

Measurement can become the work instead of enabling the work.

I have seen this. I have caused it. And it is destructive.

The warning signs: Engineers spending more time reporting productivity than being productive. Meetings about metrics instead of meetings about customer problems. Optimizing for measurement instead of optimizing for outcomes. Dashboard theater for stakeholders.

When measurement becomes performative rather than informative it is actively harmful.

The Question of Trust

You asked when do we trust that good tools in the hands of good people will create value?

My answer: I always trust that. But I also verify at a cadence that does not interfere with the work.

Trust does not mean blind faith. It means giving people the tools they need staying out of their way while they work and checking in periodically to ensure the investment is sound.

It is the same way I think about hiring: hire great people trust them to do great work measure outcomes not activities intervene only when there are problems.

The Synthesis

I think the middle ground is: Measure enough to defend investments. Not so much that measurement becomes the work.

What is enough? The minimum needed to answer these questions when budget cuts come: Are we using what we are paying for? Is it making things better? What is the business impact?

If you can answer those three questions quarterly with reasonable confidence that is probably enough.

Everything beyond that needs to justify its own existence. If a metric does not inform a decision stop tracking it.

The Final Thought

You said someone had to say this and you are right. We need voices pushing back on measurement theater.

But the answer is not no measurement. It is smart measurement that protects the investments your team needs without interfering with the creative work you do.

Thanks for the provocation Maya. It is a healthy counterbalance to the measure everything impulse. We need both perspectives to find the right balance.