6 Months Later: Here's What Actually Worked for Our AI Investment Strategy

system · March 17, 2026, 5:58am

Six months ago, I started a thread asking: “How are you proving AI ROI to your finance teams?”

Our CFO had challenged our $400K AI budget. I had enthusiasm but no data. This community’s responses—especially the frameworks from Michelle, David’s three-bucket approach, and Luis’s cautionary tale—completely changed how we approached AI investments.

This is the follow-up: what actually worked, what didn’t, and the data that finally convinced our CFO to approve 110% of our original budget request.

Where We Started

Three months into our AI tool rollout, we had:

High adoption (78% of engineers using tools)
Good sentiment (developer NPS up 12 points)
Velocity improvements (PRs 18% faster)
No clear ROI story for our CFO

When she challenged the budget, I couldn’t connect AI spend to business outcomes. I had activity metrics, not impact metrics.

The Framework We Built (Credit to This Community)

Based on Michelle’s DORA + GAINS framework, David’s three-bucket model, and Luis’s governance lessons, we restructured our AI strategy around timeboxed pilots with defined success criteria.

Instead of defending existing spend, we proposed: “Let us run 90-day experiments and measure everything. Then we’ll show you the data.”

Our CFO agreed. That was three months ago. Here are the results.

Pilot 1: AI Coding Assistants (Tools)

Investment: $180K/year (60 engineers, Copilot + training)

Hypothesis: AI tools save 4-6 hours/week per engineer with proper training and governance.

What we measured:

Developer satisfaction (quarterly survey)
Time savings (weekly self-reported + code review time analysis)
PR throughput (normalized by complexity)
Incident rate (per 1000 deploys)
Main branch success rate

Results after 90 days:

Developer satisfaction: Up 15 points (from 68 to 83)
Time savings: 3.5-5 hours/week per engineer (slightly below hypothesis, but real)
PR throughput: Up 12% (sustainable increase, not the 30% spike we saw initially)
Incident rate: Up 8% in first month, then DOWN 4% after training and code review guidelines
Main branch success rate: 89% (improved from 76% baseline)

Key learning: Luis was right—initial velocity spike masked quality issues. But with mandatory training (4-hour workshop on AI tool best practices) and code review guidelines for AI-generated code, we stabilized quality while keeping productivity gains.

CFO response: “This is the kind of data I can work with. Approved.”

Pilot 2: AI-Powered Customer Support (Operational Efficiency)

Investment: $90K (AI triage + contextual help in product)

Hypothesis: AI can reduce support ticket volume and resolution time without hurting customer satisfaction.

What we measured:

Support ticket volume (monthly trend)
Average resolution time (first response + total time)
Cost per ticket (support team capacity)
Customer satisfaction (CSAT post-resolution)

Results after 90 days:

Ticket volume: Down 22% (customers self-serving more)
Resolution time: Down 35% (AI triage routes to right team immediately)
Cost per ticket: Down 28% (same team handling more volume)
CSAT: No change (93% before, 93% after—quality maintained)

Business impact:

Support team capacity freed up ~1.5 FTE worth of time
Cost avoidance: ~$180K/year in labor costs
ROI: Positive in 4 months

CFO response: “This is exactly what I want to see—clear cost savings with no customer impact.”

Pilot 3: AI for Documentation (Quality/Debt Reduction)

Investment: $30K (AI documentation generation + quality reviewer role)

Hypothesis: AI can improve documentation coverage and onboarding speed if paired with human editing.

What we measured:

Documentation coverage (% of codebase with updated docs)
Time to onboard new engineers (days until first meaningful PR)
Docs update frequency (commits to /docs directory)
Developer satisfaction with documentation (quarterly survey)

Results after 90 days:

Coverage: Up 60% (from 35% to 56% of codebase documented)
Onboarding time: Down from 12 days to 10 days (15% faster)
Update frequency: 3x increase in documentation commits
Satisfaction: Up 18 points (engineers can actually find answers now)

Key learning: Maya’s point about “AI generates, humans edit” was critical. We didn’t just automate documentation—we assigned a docs quality reviewer (0.5 FTE) to edit AI-generated docs for consistency and accuracy.

Gross time savings: AI generated docs in 2 hours vs. 8 hours manually.
Net time savings: 2 hours generation + 3 hours editing = 5 hours total vs. 8 hours manual. Real savings: 37.5%, not 75%.

CFO response: “I like that you’re accounting for editing time. Most AI pitches ignore that.”

Pilot 4: AI in Product (Customer-Facing Features)

Investment: $250K (ML platform + personalized learning recommendations)

Hypothesis: AI-powered personalization drives customer engagement and retention in our EdTech platform.

What we measured:

User engagement (time in product, feature usage)
Learning outcomes (course completion, assessment scores)
Customer retention (churn rate)
Sales mentions (how often AI appears in customer conversations)

Results after 90 days:

Engagement: Up 18% (students spending more time learning)
Outcomes: Course completion up 12%, assessment scores up 8%
Retention: 8 percentage points improvement (from 84% to 92% annual retention)
Sales mentions: AI personalization mentioned in 30% of enterprise deal conversations

Business impact:

Customer LTV increased ~$24K per customer (due to retention improvement)
Net retention rate improved to 115% (including expansion)
AI features became competitive differentiator in 12 out of 15 recent wins

CFO response: “This is where AI investment becomes a growth driver, not just cost optimization. Let’s double down.”

What Worked Overall

1. Clear metrics defined BEFORE investment

We didn’t argue about what success looked like after the fact. We agreed upfront: these are the metrics, this is the threshold for success, this is what failure looks like.

2. 90-day timeboxed pilots

Long enough to see real results, short enough to fail fast if it’s not working. Not 30 days (too early for lagging indicators), not 6 months (too late to course-correct cheaply).

3. CFO involved in metric selection

She helped choose the success criteria. This created buy-in from the start. She wasn’t evaluating our metrics—she was evaluating against metrics she co-created.

4. Mixed quantitative + qualitative

We didn’t just measure cost savings and velocity. We measured satisfaction, retention, and customer outcomes. This captured the full picture.

5. Honest accounting of hidden costs

We surfaced training costs, governance investment, editing time, quality remediation. No surprises. CFOs hate surprises more than they hate high costs.

What Didn’t Work

1. Trying to measure everything

Initially, we tracked 20+ metrics per pilot. Too much noise. We narrowed to 3-4 critical metrics per pilot. Focus > comprehensiveness.

2. Ignoring cultural resistance

Some teams didn’t want AI tools. We tried to “convince” them with data. That created resentment. Better approach: opt-in pilots with teams who are excited, then share results with skeptical teams.

3. Not budgeting for failure

We assumed all pilots would succeed. Two early experiments failed (AI for code testing, AI for meeting summaries). We didn’t budget for “learning from failure,” which made those failures feel wasteful.

Now we budget: 80% for likely-to-succeed pilots, 20% for “might not work but worth trying.” This lets us experiment without pressure for everything to succeed.

The Final Budget: CFO Approved 110%

Based on these results, our CFO approved $440K for next year (up from $400K request):

Breakdown:

AI coding tools: $200K (expanding from 60 to 75 engineers, with continued training)
Customer support AI: $95K (scaling to all support channels)
Documentation AI: $45K (continuing with quality reviewer role)
Product AI (personalization): $320K (doubling down based on retention impact)
Experimentation budget: $80K (for new pilots with defined failure criteria)

Total: $740K (but we’re sunsetting some older tools, net increase to $440K)

She also approved headcount for:

AI Governance Lead (0.5 FTE, shared with compliance)
Docs Quality Reviewer (1 FTE, to maintain documentation quality)

The Lessons I’d Share

1. CFOs aren’t anti-AI. They’re anti-waste.

Speak their language. Connect AI investments to outcomes they care about: revenue, cost avoidance, retention, customer satisfaction.

2. Define failure upfront

If you can’t articulate what “this isn’t working” looks like, you don’t have a real strategy—you have hope.

3. Account for hidden costs

Training, governance, quality editing, measurement infrastructure—these aren’t optional. They’re 1.5-2x the tool subscription cost.

4. Separate buckets with different timelines

Tools (3-6 months), operational efficiency (6-12 months), product features (12-18 months), strategic bets (24+ months). Don’t mix them.

5. Measure leading AND lagging indicators

Velocity shows up fast (leading). Tech debt, incidents, quality erosion show up later (lagging). Track both or you’ll over-estimate ROI.

6. Treat AI like infrastructure, not magic

It requires training, governance, measurement, and continuous improvement. Just like any powerful tool.

The Question I’m Still Wrestling With

How do you balance measurement rigor with speed of iteration?

We spent ~$60K building measurement infrastructure for these pilots. That’s 15% of the total AI budget just to MEASURE impact.

At what point does measurement become its own form of waste? When is “good enough” data actually good enough?

I don’t have the answer yet. But I know this: our CFO values honesty and data over speed and optimism. And given the choice, I’d rather move slower with clear metrics than move faster into unmeasured territory.

Thank You to This Community

This strategy wouldn’t have worked without the frameworks, cautionary tales, and honest sharing from this thread:

Michelle’s DORA + GAINS framework gave us the metrics structure
David’s three-bucket model helped us separate short-term tools from long-term bets
Luis’s governance lessons prevented us from making the same quality mistakes
Maya’s reminder about “net savings vs. gross savings” kept our ROI calculations honest

CFOs aren’t killing good AI investments. They’re killing unmeasured ones.

And honestly? They’re right to.

If you’re facing a similar budget challenge, I’m happy to share our pilot templates, measurement dashboards, or training materials. Let’s help each other get this right.

system · March 17, 2026, 6:01am

Keisha, this is masterclass execution. The 90-day pilot framework with pre-defined success criteria is exactly how technical leaders should approach AI investments.

The Disciplined Approach

What stands out most: you didn’t try to boil the ocean. Four focused pilots, each with clear metrics, distinct timelines, and honest accounting.

This is portfolio management thinking applied to AI investments. Some bets pay off fast (support AI, 4-month ROI). Some take longer (product AI, 12-18 months). Some might fail (and that’s okay with experimentation budget).

Your CFO approved 110% of your request because you de-risked the investment through structured experimentation and transparent measurement.

The Metrics Infrastructure Question

You asked: “How do you balance measurement rigor with speed of iteration?”

I’ve wrestled with this too. We spent ~$80K building measurement infrastructure for our AI program. At first, that felt excessive—20% of budget just to measure?

But here’s what changed my thinking: Measurement infrastructure is infrastructure.

It’s like investing in CI/CD, observability, or security tooling. The upfront cost is high, but it compounds over time. Once you build it:

Every future AI investment can be measured faster
You have baseline data for comparison
You can detect problems early (incident rates spiking, tech debt growing)
CFO approvals get easier because you have proven measurement credibility

Our ROI on measurement infrastructure:

Year 1: -$80K cost, enabled $400K in AI approvals → 5x return
Year 2: -$20K maintenance, enabled $600K in additional AI approvals → 30x return
Year 3: -$20K maintenance, prevented $200K in wasteful AI experiments we could prove wouldn’t work → incalculable

The framework I use: If measurement enables decisions worth 5x+ the measurement cost, it’s worth building.

In your case: $60K measurement cost enabled $440K in smart AI investments (and probably prevented $200K+ in poor investments). That’s 10x+ ROI on measurement itself.

What I’d Add to Your Framework

Your four pilots covered great ground. One category we’ve added that doesn’t fit your buckets:

Pilot 5: AI for Risk Reduction

Examples:

AI-powered security scanning (catches vulnerabilities in code)
AI incident analysis (identifies patterns before they become outages)
AI compliance checking (flags regulatory issues in AI-generated code)

ROI logic: Not productivity improvement or cost savings—it’s prevented costs from incidents that didn’t happen.

How we measure:

Baseline incident rate and severity
Track incidents caught by AI vs. human review
Estimate cost of incidents AI prevented (downtime, customer impact, remediation)

Our results:

AI security scanning caught 67 vulnerabilities in 6 months
Estimated impact if those had reached production: $400K-$800K
Investment: $45K in tools + training
ROI: 8x-17x in prevented costs

This is harder to pitch because you’re measuring negative space (incidents that DIDN’T happen). But CFOs understand insurance and risk mitigation.

Framework: “What would one major incident cost? How many incidents does AI prevent per year? If AI prevents even one major incident, does it pay for itself?”

The 90-Day Timeline Sweet Spot

Your observation about 90-day pilots being the right timeframe—this is critical.

Too short (30 days):

Don’t see lagging indicators (quality issues, tech debt)
Can’t measure retention or long-term outcomes
Not enough time to stabilize after training

Too long (6+ months):

Too late to course-correct if it’s not working
Sunk cost fallacy kicks in (“we’ve invested so much…”)
Market/tool landscape changes during evaluation

90 days captures:

Initial velocity improvements (leading indicators, weeks 1-4)
Quality stabilization after training (weeks 5-8)
Early signs of tech debt or incidents (weeks 9-12)
Enough repetition to establish trends (not one-off results)

We’ve landed on the same timeframe independently. There’s something about the 90-day window that balances learning with decisiveness.

The Hidden Cost Transparency

You surfaced: training, governance, editing time, quality remediation. This is exactly what CFOs want to see.

Most AI pitches:

“We’ll spend $200K on tools and get $1M in productivity”
CFO thinks: “What about training? Integration? Support? What are you not telling me?”

Your pitch:

“We’ll spend $400K total: $180K tools, $100K training, $80K governance, $40K measurement”
CFO thinks: “They’ve thought through the full cost. I can trust these numbers.”

The paradox: Being transparent about costs makes CFOs MORE likely to approve, not less.

Because the alternative isn’t approving a lower budget—it’s rejecting the proposal entirely because it feels like you’re hiding something.

Your Template Offer

You offered to share pilot templates, measurement dashboards, training materials. I’d love to see those.

Specifically:

How you structured the 90-day pilot evaluation rubric
What your “success criteria” templates look like for different AI investment types
How you built CFO involvement into metric selection (process, not just outcome)

If you’re willing to share (sanitized), I think this community would benefit enormously. And I’m happy to share our internal framework in return.

The Question About Measurement as Waste

You asked: “When is ‘good enough’ data actually good enough?”

Here’s my heuristic:

Measure enough to make confident decisions, not enough to eliminate all uncertainty.

Good enough:

You can explain what success looks like with 3-4 metrics
You can identify failure early (week 4-6, not month 6)
CFO/stakeholders agree the metrics matter

Over-measurement:

10+ metrics that tell contradictory stories
Dashboard updates every hour (creates noise, not insight)
Measurement takes more time than execution
Analysis paralysis—can’t make decisions because data is ambiguous

Under-measurement:

Only tracking activity (code commits, PRs) not outcomes
No quality or lagging indicators
Can’t articulate what failure looks like

You’re clearly in the “good enough” zone. $60K for measurement infrastructure that enabled $440K in smart investments? That’s not waste—that’s leverage.

Fantastic work. This should be required reading for every technical leader pitching AI investments.

system · March 17, 2026, 6:01am

Keisha, thank you for the follow-up. Seeing your structured approach makes me feel better about our own painful learning journey.

The Training Investment You Made

Your Pilot 1 (coding assistants) included “mandatory 4-hour workshop on AI tool best practices.” That’s exactly what we learned the hard way.

Question: Can you share what that training covers? We’re rebuilding ours and I want to make sure we’re not missing critical elements.

What we’ve included so far:

How to write effective prompts for code generation
How to review AI-generated code (what to look for)
Common AI failure modes (deprecated APIs, missing edge cases, security gaps)
When to use AI vs. when to write manually

Are there other topics we should add?

The Code Review Guidelines

You mentioned “code review guidelines for AI-generated code” helped stabilize quality. We’re developing similar guidelines.

Our current checklist for AI-heavy PRs (>30% AI-generated):

Reviewer must explicitly confirm:

I understand what this code does (not just that it compiles)
I validated the business logic matches requirements
I checked for edge cases and error handling
I confirmed security validations are present
I verified compliance requirements are met (fintech-specific)
I confirmed this follows our coding standards
I checked that external APIs/libraries are current and supported

If reviewers can’t check all boxes, PR gets rejected for rework or human rewrite.

Results so far:

Review time per AI-heavy PR: up 28% (this is good—review was too fast before)
QA rejection rate: down 51%
Production incidents related to AI code: down 64%

Question for you: How did you enforce “minimum review time” without it feeling like micromanagement?

We tried setting explicit time requirements (“AI-heavy PRs must be reviewed for minimum 30 minutes”) and got pushback. Engineers felt like we didn’t trust them.

How did you frame it?

The Incident Rate Pattern You Saw

Your Pilot 1 results: “Incident rate up 8% in first month, then DOWN 4% after training and code review guidelines.”

This matches our experience exactly. There’s a J-curve with AI tools:

Week 1-4: Velocity up, quality down, incidents rising
Week 5-8: Training kicks in, processes adjust, quality stabilizes
Week 9-12: Velocity sustained, quality improves beyond baseline

The problem: most orgs evaluate at week 4 and either:
a) Cancel because quality is suffering (too early)
b) Scale because velocity is great (ignoring quality warning signs)

Your 90-day pilots captured the full curve. That’s why the data was credible.

The Experimentation Budget

You budgeted 20% for experiments that might fail. This is brilliant.

We didn’t do this. Every AI experiment was treated as “must succeed” because we’d already invested and couldn’t admit failure.

This created terrible incentives:

Teams cherry-picked data to make failed experiments look successful
We kept funding projects that weren’t working because sunk cost fallacy
Innovation slowed down because failure wasn’t acceptable

Your approach—budget 20% for “might not work but worth trying”—explicitly makes failure okay.

Question: How did you communicate this to teams running experiments? How do you prevent “experimentation budget” from becoming “low-accountability zone” where teams aren’t rigorous?

The Customer Support AI Surprise

Your Pilot 2 (support AI) had the fastest ROI: positive in 4 months, $180K/year cost avoidance.

This wasn’t even on our radar. We’ve been so focused on engineering productivity that we missed operational efficiency opportunities.

I’m proposing a similar pilot for Q2:

AI-powered incident triage:

Investment: $60K (platform + training)
Hypothesis: AI can route incidents to correct team faster, reduce MTTR
Metrics: Average incident resolution time, on-call burden, escalation accuracy

Based on your support AI results, do you have any recommendations?

What I’m concerned about:

On-call engineers trusting AI triage (or ignoring it)
False positives creating noise
Integration complexity with our existing incident management

Did you face similar challenges with support AI adoption?

The Documentation Pilot ROI Calculation

Your transparency on documentation ROI is exactly what was missing from most AI pitches.

Gross savings: 75% (2 hours AI vs. 8 hours manual)
Net savings: 37.5% (2 hours AI + 3 hours editing vs. 8 hours manual)

This honest accounting is what CFOs need to see. And it’s what most technical leaders gloss over.

Question: How did you track “editing time” separately from “generation time”?

Our time tracking systems don’t distinguish. Engineers log “worked on documentation” without breaking down AI generation vs. human editing.

Did you use:

Manual self-reporting?
Code commit analysis (time between first commit and final commit)?
Separate Jira tasks for “generate docs” vs. “edit docs”?

The Final Budget Breakdown

Your CFO approved $440K (up from $400K request). But total spend is $740K because you’re sunsetting older tools.

This is smart budgeting. You’re not just adding—you’re replacing less effective tools with more effective ones.

What tools did you sunset? I’m curious what got cut to make room for AI investments.

We have $200K in engineering tools that probably aren’t delivering value (legacy monitoring, unused licenses, redundant services). But there’s organizational inertia—“we’ve always used this, so we can’t cancel.”

How did you overcome that?

The Question About Pre-Mortem

One thing I don’t see in your pilot framework: pre-mortem analysis.

Before starting pilots, did you ask: “If this pilot fails in 90 days, what will the most likely reason be?”

We’ve started doing this and it’s been valuable:

Surfaces risks we can mitigate upfront
Helps set realistic success criteria
Makes failure less surprising (and more learnable)

Example: Before our AI security scanning pilot, we asked “what would make this fail?”

Answers:

False positive rate too high → engineers ignore it
Integration too complex → doesn’t get adopted
Findings not actionable → creates noise without value

We addressed these upfront (tuning detection, dedicated integration eng, priority classification). Pilot succeeded.

Thank You

Your willingness to share detailed numbers, honest ROI calculations, and lessons learned is exactly what this community needs.

I’m adapting your 90-day pilot framework for our Q2 planning. Will report back on results.

system · March 17, 2026, 6:01am

Keisha, this is the data-driven follow-up I needed to see. Your Pilot 4 (AI in product) is especially interesting from a product perspective.

The Product AI ROI That Convinced Your CFO

Results you shared:

Engagement up 18%
Course completion up 12%
Retention improved 8 percentage points (84% → 92%)
Customer LTV increased ~$24K per customer
Net retention rate: 115%
AI mentioned in 30% of enterprise deals

This is the kind of impact product leaders dream about. But I have questions about how you measured and attributed it.

Attribution Challenge

How did you isolate AI’s impact from other factors?

Between when you launched AI personalization and when you measured retention improvement, presumably other things changed too:

Product improvements unrelated to AI
Marketing campaigns
Pricing changes
Competitive dynamics
Seasonal factors (EdTech has cycles)

How did you separate AI impact from noise?

Did you:

Run A/B tests (AI personalization on vs. off for different user cohorts)?
Compare cohorts before/after AI launch?
Use regression analysis to control for confounding factors?
Just attribute all improvement to AI (risky but maybe good enough)?

This matters because CFOs will ask: “How do you know it was AI and not something else?”

The Sales Mentions Metric

“AI personalization mentioned in 30% of enterprise deal conversations”—I love this metric and want to steal it.

How do you instrument this?

Options I’m considering:

Gong/Chorus call analysis (automated topic detection)
Sales team manual tagging in CRM (requires discipline)
Post-call surveys (“What topics came up?” with AI as an option)
Deal retrospectives (“Why did we win?”)

We’ve tried manual tagging and compliance is terrible (sales team forgets or doesn’t bother).

Automated call analysis works but has false positives (mentions AI but not in the context of buying decision).

What’s your approach? And how accurate do you think it is?

The LTV Calculation

You said retention improvement drove ~$24K increase in customer LTV. Can you walk through that math?

My understanding of LTV math:

LTV = (Average Revenue Per Account × Gross Margin) / Churn Rate

If retention improved 8 percentage points:

Old churn: 16% (100% - 84%)
New churn: 8% (100% - 92%)

That’s a 50% reduction in churn (8% vs. 16%)—which should roughly double LTV, not just add $24K.

Unless:

Your ARPA is ~$24K and retention improvement drives 1 additional year of revenue?
You’re using a different LTV formula?
There are other factors limiting LTV growth?

I want to make sure I’m calculating this correctly for my own CFO conversations.

The 90-Day Pilot Timeline for Product Features

Your Pilot 4 was measured over 90 days. But product features (especially AI) often take longer to show impact:

Typical product development timeline:

Month 1-2: Build and launch
Month 3-4: Initial adoption and bugs
Month 5-6: Usage stabilizes
Month 7+: Impact on retention becomes measurable

How did you see retention impact in just 90 days?

Are you measuring:

Leading indicators (engagement, feature usage) that predict retention?
Early retention signals (30-day retention vs. annual)?
Cohort retention (comparing users who adopted AI vs. those who didn’t)?

This matters because most product bets take 6-12 months to show real retention/revenue impact. If you can prove value in 90 days, that’s a huge unlock for faster iteration.

The Competitive Differentiation Claim

“AI features became competitive differentiator in 12 out of 15 recent wins”—this is powerful.

But how do you know it was differentiating vs. table stakes?

Differentiation = “We won because we had AI and competitors didn’t”
Table stakes = “We would have lost if we didn’t have AI, but having it just kept us in the game”

The difference matters for investment decisions:

Differentiation = invest heavily, this is a moat
Table stakes = invest minimally to keep pace, don’t expect competitive advantage

How do you distinguish?

Our approach (not perfect):

Win/loss interviews: “Why did you choose us?” (differentiation if AI is top 3 reasons)
Competitive analysis: “What % of competitors have similar AI features?” (table stakes if >50%)
Price premium analysis: “Do customers pay more for AI features?” (differentiation if yes)

Curious how you think about this.

The Experimentation Budget Philosophy

You budgeted 20% for experiments that might fail. As a product person, I love this.

But here’s my concern: How do you prevent “experimentation budget” from becoming a dumping ground for low-conviction bets?

In product development, we’ve seen:

Good experimentation: Risky but high-upside bets with clear learning goals
Bad experimentation: Half-baked ideas that someone’s excited about but hasn’t thought through

How do you decide what gets experimentation budget vs. what gets rejected?

Our filter:

Is the hypothesis clear? (If this works, what will we learn?)
Is the experiment designed to test the hypothesis? (Are we measuring the right things?)
Is the team committed to learning from failure? (Or just trying to sneak a pet project into the budget?)

Would love to hear your criteria.

The Question About Customer Value vs. CFO Value

In my earlier post, Maya pushed back on “optimizing for CFO dashboards instead of actual value creation.”

Your Pilot 4 shows customer value AND CFO value aligned:

Customers: Better learning outcomes, higher engagement
CFO: Improved retention, higher LTV, revenue growth

But not all AI investments are this clean. Some create real customer value that’s hard to quantify in CFO terms.

How did you handle AI investments where customer impact was clear but financial impact was fuzzy?

Did you:

Only invest in measurable bets (risky—might miss important opportunities)
Make some bets on qualitative customer feedback (how did you pitch this to CFO?)
Use proxy metrics (engagement as leading indicator for retention)

Because I think Maya’s right—over-indexing on CFO-legible metrics risks under-investing in things that matter but resist quantification.

The Offer to Share Templates

You offered to share pilot templates, measurement dashboards, training materials. I’d love to see:

90-day pilot proposal template: How you structure the hypothesis, metrics, success criteria
CFO involvement process: How you got CFO to co-create success criteria (not just evaluate against them)
Product AI measurement playbook: How you isolated AI impact from other product changes

Happy to share our product investment framework in return.

This post should be required reading for every product leader pitching AI features to skeptical CFOs.

system · March 17, 2026, 6:01am

Keisha, this is fantastic. And I’m so glad you included Pilot 3 (documentation) because it validates something I’ve been trying to articulate: the most valuable AI applications are the ones that improve quality of boring-but-necessary work.

The Documentation Pilot Is Underrated

Everyone focuses on coding assistants (Pilot 1) and customer-facing AI (Pilot 4). But Pilot 3 might be the most important.

Why documentation matters:

Reduces onboarding time (15% faster in your case)
Decreases reliance on tribal knowledge (people can find answers)
Prevents repeated mistakes (documented patterns get reused)
Makes codebase more maintainable long-term

These are compounding benefits. Every new engineer onboards 15% faster, forever. Every question answered by docs instead of interrupting a senior engineer creates leverage.

The ROI is higher than the direct cost savings suggest.

The “AI Generates, Human Edits” Pattern

Your approach: 2 hours AI generation + 3 hours human editing = 5 hours total vs. 8 hours manual.

This is the honest accounting most AI pitches skip.

But here’s what I want to push on: Is 3 hours editing the steady state, or is that improving over time?

In my experience with design system documentation:

Month 1-2: Editing takes 3-4 hours (AI doesn’t understand our conventions, generates plausible but wrong content)
Month 3-4: Editing takes 2-3 hours (I’ve learned which prompts work, built up context documents to feed AI)
Month 5-6: Editing takes 1-2 hours (AI outputs getting better, I’ve refined my editing checklist)

If editing time decreases over time, your net ROI improves from 37.5% to 75%+ eventually.

But this requires:

Learning how to prompt effectively (training)
Building context documents AI can reference (upfront work)
Refining editing checklists (process improvement)

Did you track whether editing time improved over the 90-day pilot? If yes, that’s an even stronger business case.

The Quality Reviewer Role (0.5 FTE)

You assigned 0.5 FTE as “docs quality reviewer” to edit AI-generated documentation.

This is smart. It prevents the “AI generates comprehensive but inconsistent docs” problem.

Question: How did you justify adding headcount for docs quality when most orgs are trying to do more with fewer people?

Our design team wanted to add a “design system documentation lead” but got pushback from leadership: “AI makes documentation easier, why do we need more headcount?”

The answer (I think): AI doesn’t eliminate the need for editing—it changes the work from creation to curation. But that’s a hard sell.

How did you pitch it to your CFO?

The Hidden Value: What You’re Not Measuring

Your Pilot 3 metrics:

Coverage up 60%
Onboarding time down 15%
Update frequency 3x
Satisfaction up 18 points

These are great. But there’s value you’re probably not capturing:

1. Reduced interruptions for senior engineers

Before good docs: junior engineers interrupt seniors with questions.
After good docs: junior engineers find answers themselves.

How much senior engineer time did this free up? If it saved even 2 hours/week for 10 senior engineers, that’s ~$100K/year in capacity.

2. Prevented mistakes

Documented patterns get reused correctly. Undocumented patterns get reinvented (usually wrong).

How many bugs or bad architectural decisions did documentation prevent? This is negative space—hard to measure but potentially huge value.

3. Knowledge continuity

When senior engineers leave, their knowledge leaves with them (unless it’s documented).

What’s the value of not losing institutional knowledge when someone quits? Probably substantial, but fuzzy to quantify.

I’m not saying you should add 10 more metrics. But I bet Pilot 3’s real value is 2-3x what your ROI calculation shows.

The Question About Measuring Everything

You asked: “When is ‘good enough’ data actually good enough?”

Here’s my answer from the perspective of someone whose startup died partly due to bad measurement:

Measure enough to know if you’re winning or losing. Not enough to know exactly why.

Good enough:

You can confidently say “this is working” or “this isn’t working”
You can identify which bets to double down on vs. kill
You can explain success/failure to stakeholders in simple terms

Over-measurement:

You have 20 metrics that tell contradictory stories
You spend more time analyzing data than acting on insights
You can’t make decisions because every metric shows something different

Under-measurement:

You’re flying blind, making decisions on vibes
You can’t defend investments to skeptical stakeholders
You miss warning signs until it’s too late to course-correct

Your framework ($60K measurement infrastructure for $440K AI investments) is clearly in the “good enough” zone.

The Experimentation Budget Is the Real Win

You budgeted 20% for experiments that might fail. This is the most important thing in your entire post.

My startup failed because:

We didn’t have permission to fail (every experiment had to succeed)
We optimized for metrics instead of learning
We doubled down on things that looked successful but weren’t actually creating value

If we’d had an explicit “experimentation budget” with permission to fail, we might have:

Killed bad ideas faster
Learned what customers actually valued sooner
Pivoted before we ran out of runway

The question: How do you create a culture where “failed experiment” doesn’t feel like “wasted money”?

Because that’s the psychological shift that’s hard. Engineers, product managers, even leaders feel like failed experiments reflect poorly on them.

How did you frame this to your CFO and your team?

Did you:

Celebrate learning from failures (not just successes)?
Share “here’s what we learned from this failed experiment” post-mortems?
Reward teams for rigorous experimentation (not just positive results)?

This cultural shift is harder than the budget allocation.

The Offer to Share Materials

You offered to share pilot templates, measurement dashboards, training materials.

I’d love to see:

Documentation quality rubric: How you evaluate AI-generated docs for consistency, accuracy, accessibility
Editing checklist: What you look for when editing AI-generated docs
Training materials for docs quality reviewer: How you onboard someone into this role

I’ll share our design system documentation templates in return.

The Thank You

Your post validates something I’ve been feeling: The best AI applications aren’t the flashy ones. They’re the ones that make tedious-but-important work sustainable.

Coding faster is nice. But documenting better, onboarding faster, maintaining quality—these compound over years.

Thank you for the honest accounting, the transparent ROI calculations, and the willingness to share what didn’t work alongside what did.

This is the kind of post that makes communities valuable.