Skills-Based Hiring Is Replacing Degree Requirements—But Are We Just Shifting Gatekeeping From Universities to LeetCode?

General
1

We just wrapped our Q1 hiring review, and I noticed something that’s been bothering me. At our Series B fintech startup, we proudly removed degree requirements from all engineering job descriptions last year—a move I championed to our board as “expanding the talent funnel.”

The data looked great on paper: 87% of hiring managers are moving toward skills-based hiring, and companies that removed degree requirements saw their talent pipeline expand by 8.2x in AI roles. We were following the trend, doing the right thing.

But here’s what actually happened: Our candidate pool didn’t get more diverse. It got different.

Instead of filtering for “Computer Science degree from a top university,” we’re now filtering for “completes our 4-hour take-home assignment” and “solves 3 LeetCode mediums in 90 minutes.” We traded one gatekeeping mechanism for another.

The Promise vs. The Reality :bar_chart:

The Promise: Skills-based hiring democratizes access to tech careers. No more arbitrary degree requirements that exclude talented people who couldn’t afford college or chose alternative paths.

The Reality: We replaced credential gatekeeping with time and access gatekeeping.

Who has 20-30 hours to practice LeetCode problems? Not the working parent balancing childcare. Not the person working two jobs to pay rent. Not the career switcher already spending nights learning to code.

Meanwhile, the 22-year-old new grad with minimal obligations? They have evenings and weekends to grind through 300 algorithm problems.

Three Types of Gatekeeping :construction:

I’ve been thinking about this in terms of a framework:

1. Credential Gatekeeping (the old problem)

  • Requires formal degrees, brand-name schools, prestigious internships
  • Explicitly excludes people without access to traditional education
  • We thought we solved this

2. Time/Access Gatekeeping (the new problem)

  • Requires significant unpaid preparation time
  • Favors candidates with fewer obligations and more resources
  • We accidentally created this

3. Definition Gatekeeping (the persistent problem)

  • Implicitly defines “skill” as what we’re comfortable measuring
  • LeetCode measures algorithmic problem-solving, not collaboration, communication, debugging, or shipping features
  • 78% of developers say these assessments don’t match real-world work
  • We’re still stuck here

The Strategic Question :bullseye:

I’m not arguing we should go back to degree requirements. I’m asking: What are we actually trying to measure, and does our assessment method predict job success?

If we’re testing for problem-solving ability, does solving tree traversal problems correlate with shipping features customers want? If we’re testing for technical competency, does a 4-hour take-home predict how someone collaborates in a real codebase?

At our company, we don’t have good answers yet. We know our current approach isn’t working—our hire-to-promote rate hasn’t improved since removing degree requirements, and our engineering diversity numbers are flat.

What I’m Curious About :light_bulb:

For those of you hiring engineering teams:

  1. Have you actually validated that your skills-based assessments predict job performance better than the degree requirements they replaced?

  2. Who are you accidentally filtering out when you require extensive technical interview prep?

  3. What alternatives are working? Project portfolios? Pair programming? Trial projects?

I suspect we’re all figuring this out in real-time, and I’d love to hear what’s actually working vs. what just sounds good in principle.

The cynical take: We didn’t make hiring more equitable. We just shifted which 22-year-old gets in—from the one with a Stanford degree to the one who spent 6 months grinding LeetCode.

The optimistic take: We’re in the messy middle of a real transition, and if we’re intentional about measurement and access, skills-based hiring can work.

Which is it? I honestly don’t know yet.

2

This hits close to home, @product_david. I’m dealing with this exact tension at my EdTech startup right now, and the equity dimension you’re highlighting is real.

What Happened When We Removed Degree Requirements :light_bulb:

18 months ago, we proudly announced we were dropping degree requirements for all engineering roles. The goal: expand our pipeline, especially for candidates from underrepresented backgrounds who face systemic barriers to traditional CS degrees.

What we expected: More diverse candidate pool, especially Black and Latino engineers, career switchers, self-taught developers.

What actually happened: Our pipeline got bigger but less diverse. The percentage of candidates from underrepresented backgrounds actually decreased by 12 percentage points.

We were confused until we dug into the data. Turns out, the assessment itself became the new barrier.

The Time Barrier Is Real :bullseye:

Our standard process:

  • 1-hour phone screen (unpaid)
  • 4-hour take-home coding challenge (unpaid)
  • 3-hour onsite with algorithm whiteboarding (unpaid)
  • Total: ~8 hours of unpaid work

Who can afford this?

Not the single parent working full-time who barely has childcare coverage for the onsite interview. Not the person working two jobs trying to break into tech. Not the career switcher who already spent 6 months on nights and weekends learning to code.

But the 23-year-old software engineer who just quit their job and has severance? They can do your 4-hour take-home and 10 other companies’ take-homes in the same week.

The data backs this up: Research shows that skills-based hiring can expand talent pools by 8.2x, but only if the assessments themselves are accessible. Ours weren’t.

Assessment Design Matters :heart:

Here’s what we changed:

Before:

  • Algorithmic challenges requiring 20-30 hours of LeetCode prep
  • Timed assessments with no accommodations

After:

  • Practical, job-relevant project: “Here’s a bug in our actual codebase—debug and fix it”
  • Untimed (48-hour window, but realistically 2-3 hours of work)
  • Explicit accommodations offered upfront
  • Paid (we compensate for take-home time at $50/hour)

Results so far (6 months in):

  • 23% increase in candidates from underrepresented backgrounds completing the process
  • Hire-to-performance correlation improved (we actually measured this)
  • Candidates report better experience (NPS up 18 points)

It’s not perfect, but it’s directionally better.

The Uncomfortable Questions We’re Still Asking

  1. Are we measuring the right “skills”? Our job is building educational technology—collaboration, empathy for users, and debugging real-world issues matter more than inverting binary trees. Are we testing for that?

  2. Who still can’t access our process? Even with paid take-homes, we’re excluding people who can’t take 3 hours of PTO for an onsite interview. Remote-first interviewing helps, but it’s not enough.

  3. What’s the actual validity? We’re tracking hire-to-performance correlation quarterly now. If our assessment doesn’t predict success, it’s just gatekeeping with extra steps.

Call to Action :bullseye:

I’d love to see more companies publish assessment equity audits:

  • What % of candidates from different backgrounds complete each stage?
  • Where do people drop off, and why?
  • Does assessment performance actually correlate with job performance 12 months later?

Without this data, “skills-based hiring” is just a buzzword. We’re not making hiring more equitable—we’re just moving the barriers around.

@product_david your framework is spot-on. The question isn’t just “are we measuring skills instead of credentials?” It’s “are we measuring the right skills in a way that’s accessible to the people we want to hire?”

If the answer is no, we’re just doing gatekeeping with better PR.

3

Both of you are raising critical points, and I want to add the perspective from a large organization where scale creates different constraints. At my Fortune 500 financial services company, we’re wrestling with this exact tension.

The Volume Problem Is Real :thinking:

Context: We typically get 400-600 applications per engineering role. Even for junior positions.

When @vp_eng_keisha talks about 8 hours of assessment time per candidate, multiply that by 500 candidates, and you’d need a team of people just reviewing assessments full-time. We don’t have that luxury—our engineering managers are already stretched thin.

So I’m sympathetic to @product_david’s critique about gatekeeping, but I also need to be honest: at scale, some form of filtering is unavoidable. The question is: which filter causes the least harm while still being operationally feasible?

What We’re Trying (Imperfectly) :thought_balloon:

We moved to a tiered approach based on role level:

Junior Engineers (0-3 years):

  • Short (45 min) practical debugging exercise, not algorithms
  • Focus: Can they read code, understand it, and make a reasonable change?
  • No LeetCode prep required

Mid-Level Engineers (3-7 years):

  • 2-hour take-home: “Add a feature to this existing codebase”
  • Optional: Can submit a portfolio project instead
  • We pay $100 for completed take-homes (regardless of outcome)

Senior+ Engineers (7+ years):

  • System design conversation (no whiteboarding)
  • Architecture review: “Here’s a scaling problem we actually faced—how would you approach it?”
  • Reference checks weighted heavily

Results after 12 months:

  • 34% increase in candidates from non-traditional backgrounds making it past initial screen
  • Hire-to-performance: too early to tell definitively, but early indicators are positive
  • Time-to-hire increased by 8 days on average (board is not thrilled)

The Uncomfortable Truth :balance_scale:

Here’s where I struggle: Even with this approach, we’re still gatekeeping. We’re still asking people to spend unpaid time (even $100 doesn’t cover the actual opportunity cost). We’re still filtering for people who can take time off work for interviews.

The reality at scale is that perfect equity is in tension with operational efficiency. I don’t like this trade-off, but pretending it doesn’t exist doesn’t help either.

Questions I’m Still Wrestling With :bullseye:

  1. What’s the right balance between efficiency and equity? Keisha’s approach (paid take-homes, accommodations, job-relevant assessments) is clearly more equitable, but requires significant resources. Not every company can afford this, especially smaller startups. Does that mean skills-based hiring only works at well-funded companies?

  2. How do we measure assessment validity at scale? We’re trying to track hire-to-performance, but with long feedback loops (12-18 months) and confounding variables (manager quality, team fit, etc.), it’s hard to isolate the impact of the assessment itself.

  3. Are we still gatekeeping on proxy measures? Our “practical debugging exercise” still favors people with professional coding experience over self-taught developers who might have gaps in their knowledge but tremendous learning ability. Are we just trading one credential for another?

A Proposal: Transparency About Trade-offs :bar_chart:

Maybe the answer isn’t finding the perfect assessment—maybe it’s being honest about the trade-offs we’re making.

Instead of claiming “we’re skills-based now, we’re equitable,” what if companies published:

  • Drop-off rates at each stage by demographic group
  • Time/cost burden on candidates
  • Validation data: Does assessment performance correlate with job success?
  • What trade-offs we’re making (efficiency vs. equity, cost vs. quality, etc.)

At least then candidates could make informed choices about where to apply, and we could learn from each other what actually works.

I don’t have perfect answers, but I appreciate this conversation. @product_david’s framework helps—we need to be intentional about which gatekeeping mechanisms we choose, because we can’t eliminate all of them at scale.

The question is: Are we choosing the ones that align with our actual goals (finding great engineers) or the ones that are just operationally convenient?