We Built a Documentation ROI Dashboard - Here's What Actually Moved the Needle (And What Didn't)

[eng_director_luis]

Following up on the documentation ROI discussion—I want to share our actual implementation journey because the theory is one thing, but the reality of building a docs metrics dashboard taught us some unexpected lessons.

Context: Financial Services, 40-Person Engineering Team

We’re a Fortune 500 financial services company with legacy systems, compliance requirements, and distributed teams. Documentation was… let’s call it “organically grown” (read: chaotic). Some teams had amazing docs, others had nothing.

Six months ago, I pitched leadership on a documentation metrics initiative. Got approval for a 3-month pilot. Here’s what we built and what we learned.

What We Built

Tech Stack:

• Documentation: Confluence (already had it)
• Ticket tracking: Jira Service Management
• Product analytics: Pendo
• Custom dashboard: Grafana + PostgreSQL

Metrics We Tracked:

1. Article Views & Search Queries: Basic engagement
2. Ticket Deflection Rate: (Docs viewed before ticket filed) / (Total tickets)
3. Search Failure Rate: Searches with no results clicked
4. Time to Answer: Time from doc page load to problem resolution (proxy: no ticket filed within 24h)
5. Onboarding Completion Time: Days from account creation to first successful transaction

The Surprising Finding That Changed Our Approach

Our most-viewed documentation wasn’t reducing tickets.

Top 10 most-viewed pages: Authentication, API rate limits, Error code reference, Database schema, Payment processing.

Support tickets about those exact topics: Still the majority of our queue.

Why? After analyzing user sessions (with Pendo session replay), we found:

• Users opened the right documentation page
• Spent 3-5 minutes reading
• Couldn’t find the specific answer they needed
• Gave up and filed a ticket

The docs were comprehensive—too comprehensive. 2,000-word pages with every edge case. Users got lost trying to find “how do I do X” buried in paragraph 14.

What Actually Moved the Needle

1. Search Failure Tracking

We started logging unsuccessful searches (query + no result clicked + ticket filed within 24 hours). Those queries became our documentation backlog.

Example: “How to retry failed payment” was searched 47 times in one month with 0 result clicks and 38 subsequent tickets. We wrote a targeted 200-word doc specifically for that query. Tickets dropped 85%.

2. “Time to Answer” Over “Page Views”

Shifted from measuring engagement to measuring effectiveness. How long from “user opens docs” to “user solves problem”?

We proxy “problem solved” as “no support ticket filed within 24 hours after viewing docs.” Not perfect, but directionally correct.

3. Onboarding Milestone Tracking

For new customer implementations, we tracked time to complete each step (account setup → first API call → first transaction → production deployment).

Before measurement: 8 days average
After targeted docs improvements: 5 days average

The Real Results (6 Months In)

• Deflection rate: 28% (up from ~15% baseline)
• Support cost savings: $200K+ annually (2.5 FTE support engineers worth of tickets deflected)
• Onboarding acceleration: 37.5% faster (8 days → 5 days)
• Search success rate: 68% (up from 41%)

We used these numbers to justify hiring a full-time technical writer. Approved within one budget cycle.

The Uncomfortable Trade-off: Speed vs. Depth

Here’s the part that keeps me up at night:

Once we started measuring “time to answer,” some engineers optimized by writing shorter docs. Which technically worked—users found answers faster! The metric improved!

But we noticed a secondary pattern: repeat questions from the same users increased.

Someone would solve their immediate problem (deflected ticket! ), but a week later they’d be back with a related question because they never understood the underlying system.

We were optimizing for short-term problem-solving at the expense of long-term knowledge building.

Current Hypothesis: Track “Repeat Question Rate”

We’re now tracking how many users file multiple related tickets within 30 days. If someone files 5 tickets about payment processing in a month, our docs aren’t teaching—they’re just answering point queries.

Early data suggests docs that prioritize conceptual understanding (even if longer) have lower repeat question rates, even if initial “time to answer” is slower.

Are we measuring this correctly? How do other teams handle the quality vs. speed trade-off?

Open Questions for the Community

1. How do you measure documentation quality vs. just quantity/speed?
2. What metrics predict long-term knowledge transfer, not just short-term deflection?
3. Has anyone successfully tracked “mental model building” in a quantitative way?
4. For those with doc dashboards: What metrics did you add/remove after the first 6 months?

I’m convinced measurement is necessary to justify investment, but I’m still figuring out what to measure to optimize for real user understanding, not just ticket reduction.

Would love to hear from others who’ve walked this path—especially the mistakes you made that we can avoid.

[product_david]

Luis! This is SO helpful. The session replay insight is genius—I never would have thought to watch users actually navigate the docs.

The Information Architecture Problem You’re Describing

As someone who thinks about design systems and user experience constantly, what you’re seeing is a classic IA (information architecture) failure disguised as a content problem.

Your users knew which doc to open (authentication, API rate limits, etc.), which means your high-level navigation worked. But they couldn’t find the specific answer within that page, which means your page-level structure failed them.

This is the documentation equivalent of a component library with great top-level organization but terrible prop documentation. “I know I need the Button component… but which of these 47 props do I actually need?”

Design Solutions for Your “Too Comprehensive” Problem

Some ideas from the UX world that might help:

1. Progressive Disclosure
Instead of one 2,000-word page, structure it as:

• Quick Start (200 words): “99% of users need this”
• Common Use Cases (expandable sections)
• Advanced / Edge Cases (collapsed by default or separate page)

Users get fast answers, but depth is available when needed.

2. Task-Based Navigation
Your search failure data is GOLD. “How to retry failed payment” tells you users think in terms of tasks, not concepts.

Instead of organizing docs by technical architecture (what engineers think about), organize by user goals (what users are trying to do).

3. Visual Hierarchy & Scanability
If users are spending 3-5 minutes on a page and still missing the answer, the visual design might be working against them. Are you using:

• Clear headings with jump links?
• Code examples before prose explanations?
• Callout boxes for “most common question”?
• Visual markers for “start here” vs “advanced”?

Your “Repeat Question” Metric Is Perfect

Tracking whether users file multiple related tickets is exactly the quality measure you need. You’re measuring “did we teach them to fish, or just give them a fish?”

This is way better than NPS or satisfaction surveys, which measure feeling, not learning.

Question About Your Dashboard

You mentioned Grafana + PostgreSQL for the custom dashboard. How are you connecting Confluence, Jira, and Pendo data? Are you:

• Using their APIs to ETL into Postgres?
• Using webhooks for real-time updates?
• Batch syncing daily?

Asking because I’m considering building something similar for our design system docs and want to understand the integration complexity.

Also: Did you build the session replay analysis manually, or did you automate pattern detection somehow?

[carlos_ml]

Luis, impressive implementation. The “time to answer” metric is clever—I’m stealing that framework for our product analytics.

Questions About Your Methodology

1. How do you measure “Time to Answer”?

You said: “Time from doc page load to problem resolution (proxy: no ticket filed within 24h)”

But how do you actually calculate that? Are you:

• Tracking when they close the browser tab?
• Measuring time until they leave the docs site?
• Using some Pendo event to mark “success”?

I’m curious because attribution is always the hardest part. If someone reads docs at 2pm and doesn’t file a ticket for 24 hours, did the docs work? Or did they solve it some other way (Slack, Stack Overflow, figured it out independently)?

2. What’s Your Tooling Stack Cost?

You mentioned Grafana + Pendo + custom PostgreSQL setup. What’s the all-in cost for:

• Pendo licensing
• Engineering time to build/maintain the dashboard
• Ongoing operational costs

Asking because when I pitch this to finance, they’ll want to know if the $200K support savings is net of dashboard costs.

3. Confounding Variables in Onboarding

8 days → 5 days for onboarding is great. But how did you isolate that documentation was the driver vs:

• Product UX improvements?
• Better customer success onboarding calls?
• Simpler customer profiles (smaller companies vs enterprises)?

Not questioning the result—just want to understand attribution methodology for when my CFO inevitably asks “how do you know it was the docs?”

The Business Case Angle

Your results ($200K savings, 37.5% faster onboarding) are exactly what finance teams need to hear. But I’d add one more metric to your dashboard:

Revenue Impact of Onboarding Speed

If your average new customer is worth $X ARR and you’re onboarding Y customers per quarter, cutting 3 days off onboarding means:

• Revenue recognized 3 days earlier
• Faster time to expansion opportunities
• Potentially higher close rates (if prospects evaluate during trial and faster onboarding = better trial experience)

Have you quantified the revenue upside, or are you focused primarily on the cost reduction story?

[customer_dev]

Luis, this is a great case study. One strategic question as you think about scaling this:

What Happens When Your Team Grows 3X?

You’re at 40 engineers now. Let’s say you grow to 120 engineers over the next 18 months (common trajectory for Series B/C companies).

Does your documentation dashboard model scale, or does it break?

I’m thinking about:

Metric Decay: Your 28% deflection rate is based on current team size, documentation scope, and product complexity. As you add features, does deflection naturally decrease because there’s more surface area to document?

Dashboard Maintenance: Who owns keeping Confluence/Jira/Pendo integrations running? One engineer? A platform team? What happens when that person leaves?

Documentation Velocity: Can one technical writer keep up with 120 engineers shipping features? How do you prevent docs from becoming stale (which would tank your deflection metrics)?

Organizational Buy-In: Right now you’re probably the champion driving this. What’s the succession plan? How do you institutionalize documentation measurement so it survives leadership changes?

The Build vs. Buy Question

You built custom dashboards with Grafana + Postgres. That works at 40 people, but have you evaluated whether commercial documentation platforms (Readme, GitBook, Archbee, Document360) with built-in analytics might be better long-term?

I’m not advocating one way or the other—just asking whether the ROI of building custom tooling holds up as complexity increases.

At my company we built custom internal platforms for 5+ years, then realized commercial solutions had matured enough that we were spending more maintaining our custom tools than we’d save vs. buying SaaS.

Has that inflection point arrived for documentation dashboards, or is custom still the right call?

[diana_data]

Luis, I want to dig into the cultural impact of your dashboard—did measuring documentation change how your engineers actually write docs?

The Behavior Change Question

You mentioned engineers started writing shorter docs to optimize “time to answer.” That’s metric gaming, but it’s also a signal that your team is paying attention to the dashboard.

Which raises interesting questions:

1. Did Documentation Become Part of Definition of Done?

Before the dashboard, was documentation optional? After you started showing deflection metrics in monthly reviews, did teams start treating docs as a required deliverable?

If so, that’s a huge cultural shift. But was it positive (engineers care about docs now) or negative (engineers write minimum viable docs to check a box)?

2. How Do You Handle Documentation Ownership?

When a feature ships, who’s responsible for writing docs?

• The engineer who built it?
• The product manager who spec’d it?
• A dedicated technical writer?
• Customer success team?

And when documentation is outdated (which your dashboard presumably surfaces via increased tickets), who fixes it? Original author? On-call? The tech writer?

3. Did Metrics Create Accountability or Blame?

This is the part I worry about most. When a team has a low deflection rate, is that:

• Accountability: “Let’s improve our docs”
• Blame: “Why is your team’s documentation worse than others?”

If it becomes a blame metric, people game it. If it’s an improvement metric, people engage with it constructively.

How did you navigate that leadership balance?

What I’m Really Asking

I manage a 75-person engineering org. I’ve considered implementing similar documentation metrics, but I’m worried about unintended consequences:

• Engineers spending time optimizing docs for the metric instead of for users
• Documentation becoming a checkbox compliance exercise
• Team comparisons creating competition instead of collaboration

Did you experience any of these? How did you course-correct when metrics started driving wrong behaviors?

Your “repeat question” metric is smart specifically because it’s harder to game—you can’t fake whether users actually understood vs. just found a quick answer.

But I’m curious: when you saw engineers writing shorter docs and repeat questions increasing, how did you communicate that as a leadership issue without making it feel punitive?