We Lost $2M in Regulatory Fines Because Our Compliance Lead Retired: A Framework for Critical Knowledge Audits

General
1

Three years ago, our senior compliance engineer retired after 15 years with the company. She’d built most of our regulatory reporting systems, knew every audit requirement, understood the exceptions to every rule.

We threw her a great retirement party. We did exit interviews. We had a 2-week overlap with her replacement.

Three months later, we received a $2M fine from our primary regulator for failing to properly report cross-border transactions. The requirement was documented—buried in page 147 of a 200-page process manual that nobody had opened in years.

That was my wake-up call.

The Framework We Built

I had to answer two questions for our executive team:

  1. How do we prevent this from happening again?
  2. How do we systematically identify and mitigate knowledge risk?

Here’s the framework we developed. It’s saved us from at least 3 other potential regulatory issues, and it’s cut our knowledge transfer risk by about 60%.

Step 1: Critical Knowledge Audit

We built a matrix: People × Systems × Risk

For each critical system, we asked:

  • Who knows how it works? (Primary, secondary, tertiary)
  • What’s the bus factor? (How many people need to be gone before we’re in trouble?)
  • What’s the regulatory/business risk if it breaks?
  • How documented is it? (1-5 scale)

This is time-consuming. We spent 40 hours just mapping our top 30 systems. But the output was eye-opening.

Example:

System Primary Secondary Tertiary Bus Factor Risk Level Doc Quality
Cross-border reporting Sarah (retiring) None None 1 Critical 2/5
Payment processing Mike, Jordan Alex, Sam None 2 High 3/5
Customer onboarding Team knowledge - - 5+ Medium 4/5

Just seeing it in a table made the risk visceral for our executives.

Step 2: Triage Framework

You can’t document everything. We prioritize based on two dimensions:

Immediate action (next 30 days):

  • Bus factor = 1 AND risk = Critical → RED ALERT
  • Bus factor ≤ 2 AND risk = High → High priority

Monitor closely:

  • Bus factor ≤ 2 AND risk = Medium → Medium priority
  • Bus factor ≥ 3 regardless of risk → Watch but don’t panic

For our compliance lead’s retirement, cross-border reporting was a RED ALERT that we completely missed.

Step 3: Knowledge Transfer Plan Template

For each high-priority item, we create a structured transfer plan:

1. Document the why, not just the what:

  • Decision history (why did we build it this way?)
  • Failed approaches (what did we try that didn’t work?)
  • Regulatory context (what requirements drive this?)
  • Edge cases and exceptions (the stuff that’s not obvious)

2. Create redundancy:

  • Shadow the expert (1-2 people spend significant time learning)
  • Knowledge-sharing sessions (recorded, searchable)
  • Pair on maintenance tasks
  • Cross-train on adjacent systems

3. Test the transfer:

  • Can the secondary owner handle a production incident alone?
  • Can they explain it to a new hire?
  • Can they make a decision without consulting the primary?

4. Maintain the knowledge:

  • Quarterly review of critical system docs
  • Rotate ownership every 18-24 months
  • New hires touch critical systems within first 90 days

Step 4: Success Metrics

We measure:

  • Bus factor improvement: Average bus factor increased from 1.8 to 3.2 for critical systems
  • Documentation coverage: Critical systems at 4+/5 documentation quality
  • Knowledge distribution: Number of people who can independently operate each system
  • Incident response: Mean time to engage subject matter expert (want this to go DOWN as docs improve)

The 60% Risk Reduction

After 18 months of following this framework:

  • Zero regulatory issues related to knowledge gaps
  • Average bus factor for critical systems increased by 78%
  • Onboarding time for engineers cut from 6 months to 3.5 months
  • Two unplanned departures (resignations) had minimal impact

The framework isn’t perfect, but it’s systematic rather than reactive.

Warning: Don’t Wait for Exit Interviews

The biggest mistake we made with our compliance lead was assuming 2 weeks of overlap was enough. By the time someone gives notice, you’re already behind.

Start the audit now. Identify your knowledge risks before they become knowledge crises.

What critical knowledge is walking around in someone’s head in your organization right now? And what happens if they give notice tomorrow?

Resources:

2

Luis, I love this framework! :light_bulb: But I’m trying to figure out how to apply it outside of engineering and compliance contexts.

In design systems work, so much of the knowledge is tacit—it’s not in the code or the docs, it’s in the accumulated judgment about:

  • Why this component has these constraints (user research findings from 2 years ago)
  • Which design patterns we tried and abandoned (and why)
  • How different teams have extended the system (and which extensions should become official)
  • The trade-offs we made between flexibility and consistency

The Bus Factor for Design Knowledge

When I do your People × Systems × Risk audit for design:

Knowledge Area Primary Secondary Bus Factor Risk Doc Quality
Design token rationale Me None 1 High 1/5
Accessibility patterns Sarah Me 2 Critical 3/5
Component API decisions Me Jordan 2 Medium 2/5

Yep. I’m the single point of failure for core design decisions. :grimacing:

The Challenge with Design Documentation

Here’s what makes this hard in design/product contexts:

1. The “why” is often user research and iteration:

  • We tried 5 different navigation patterns
  • User testing showed pattern #3 was most intuitive
  • But we shipped pattern #4 because of technical constraints
  • → How do I document that context without 20-page essays?

2. Decisions are visual and interactive:

  • Written documentation doesn’t capture “this feels right”
  • The judgment that comes from seeing 1000 iterations
  • Knowing which rules to break and when

3. Design rationale lives in Figma comments, Slack threads, Loom videos:

  • Scattered across tools
  • Timestamped to specific moments
  • Not searchable or organized

What I’m Trying

I’ve started experimenting with:

  • Decision log in Figma: Every major component has a “decisions” section explaining the why
  • Video walkthroughs: 5-minute Loom explaining the thinking behind each pattern
  • Regular “design archaeology” sessions: New designers pair with me to understand old decisions
  • User research repository: Organizing insights by theme, not by study

But honestly? I’m not confident it would survive my departure.

Question for the group: How do you document expertise that’s more about judgment and taste than process and procedure?

3

This is exactly the kind of framework I need. Two questions about scalability:

1. How do you maintain this as you grow rapidly?

We’re scaling from 50 to 120 engineers this year. Your framework assumes relatively stable systems and teams. But when you’re:

  • Launching new systems monthly
  • Reorganizing teams quarterly
  • Hiring senior engineers who become new “primary owners”

…doesn’t the audit become stale almost immediately?

We tried something similar 18 months ago. The matrix was out of date within 3 months because of how fast we were moving.

2. Can this be partially automated?

I’m thinking:

  • Code ownership tracking: GitHub already knows who touches what files
  • On-call rotations: PagerDuty knows who responds to which incidents
  • Documentation coverage: Automated scanning for systems without docs
  • Knowledge graphs: Tools like Glean or Guru that map who knows what

Has anyone successfully automated the “bus factor audit”? Or does it have to be manual to be accurate?

The Scale Challenge

At 50 engineers, doing a 40-hour audit for 30 systems is manageable. At 120 engineers with 80+ systems, that’s a full-time role just maintaining the matrix.

I’m not pushing back on the value—I completely buy the ROI. I’m asking: How does this scale without becoming its own operational burden?

Maybe the answer is: “It’s worth dedicating a person to this.” But I need to make that case to the board, and “we need an FTE just to track who knows what” is a hard sell.

@eng_director_luis - did you assign someone to own this process? Or is it distributed responsibility?

4

Love this framework, Luis. It bridges engineering and product beautifully.

@maya_builds - your design knowledge question resonates. We have the same challenge in product:

The Product Knowledge Gap

Product documentation is often even worse than engineering docs because we assume “everyone knows why we built this.”

Our bus factor audit for product knowledge looked like:

Knowledge Area Primary Risk Level Documented?
Why we pivoted from SMB to enterprise Me Critical No
Customer segment prioritization rationale Me High Partially
Pricing model history and failed experiments Former PM (left 6 months ago) Critical No
Competitive positioning decisions Me + Marketing Medium Yes

The pricing knowledge was completely lost. We had to reverse-engineer our own decisions from old Slack threads and customer interviews.

Cross-Functional Documentation Problem

Here’s what makes this harder across functions:

Engineering documents how but not why we chose this approach

  • Code reviews focus on implementation, not business context
  • Technical docs explain what the system does, not what customer problem it solves

Product documents what but not how we validated it

  • PRDs describe features, not the research/iteration that led there
  • We don’t capture the 10 ideas we rejected to get to the 1 we shipped

Design documents why but scattered across too many tools

  • Figma files, Loom videos, user research repos, Slack threads
  • Not discoverable, not version-controlled, not searchable

A Cross-Functional Twist on Luis’s Framework

What if the knowledge audit included cross-functional context?

For each critical feature/system:

  • Engineering: How it works, technical decisions, trade-offs
  • Product: Why we built it, what customer problems it solves, what we learned
  • Design: What we tried, what user research showed, what patterns we established

This forces us to think about knowledge transfer holistically, not just within functions.

Question: Has anyone successfully built shared documentation practices across engineering, product, and design? Or do they always end up siloed?

5

This framework is solid, Luis. What I’d add: Onboarding is your documentation validation system.

We’re scaling fast (25 → 80+ engineers), and here’s what we’ve learned:

New Hires Test Your Documentation

Every new engineer who joins reveals gaps in your knowledge systems. We formalized this:

Week 1-2 Onboarding Task:
“Document everything you couldn’t figure out from existing docs.”

The output is gold:

  • Which systems are completely undocumented
  • Which docs are out of date (new hire follows them, breaks things)
  • Which docs exist but are unfindable
  • Which tribal knowledge gets shared verbally instead of written down

This creates a feedback loop: New hire → identifies gap → we fix it → next new hire has better docs → repeat

Tying This Back to Bus Factor

Your matrix could include a column: “Can a new hire understand this independently?”

System Primary Bus Factor New Hire Independence Risk
Auth system Jordan 1 No - requires 5+ Slack questions High
Payment processing Team 3+ Partial - docs exist but incomplete Medium
User notifications Sarah, Mike 2 Yes - good docs + runbooks Low

The “New Hire Independence” metric is leading indicator of bus factor risk. If new hires can’t figure it out alone, you have undocumented tribal knowledge.

To @cto_michelle’s Automation Question

We partially automated this:

1. Documentation coverage bot:

  • Scans repos for README files, wiki links, ADR presence
  • Flags systems with no documentation
  • Weekly Slack summary of coverage gaps

2. Onboarding friction tracker:

  • New hires tag Slack questions with #onboarding
  • We aggregate: Which systems generate the most questions?
  • Those are documentation priorities

3. Quarterly knowledge audit:

  • Yes, it’s manual
  • But we only deep-audit systems that triggered alerts (no docs, high question volume, critical risk)
  • Reduces scope from “audit everything” to “audit the red flags”

This isn’t as comprehensive as Luis’s full matrix, but it’s scalable and catches 80% of the risk with 20% of the effort.

Culture Piece

The other thing: We promote people partially based on knowledge sharing.

  • IC → Senior IC: “Have you documented your expertise?”
  • Senior IC → Staff: “Have you made others experts in your domain?”
  • Staff → Principal: “Have you eliminated single points of failure in critical areas?”

When knowledge sharing is a promotion criterion, people actually do it.

@eng_director_luis - curious how you incentivized compliance with the framework. Was it top-down mandate or cultural shift?