🏛️ SF Tech Week Day 1: AI Regulation Reality Check - What Enterprises Are Actually Facing

General
4

Reading this thread while sitting in the SF Tech Week “AI Infrastructure at Scale” session, and I’m nodding so hard my neck hurts.

The Hidden Infrastructure Costs Nobody Talks About

The speaker (Head of ML Infrastructure at Stripe) just broke down their actual AI infrastructure costs:

For a single production ML model serving 10M requests/day:

Initial estimate (from POC):

  • Compute: $5K/month
  • Storage: $500/month
  • Total: $5.5K/month

Actual production costs:

  • Inference compute: $45K/month (9x higher)
  • Training compute: $25K/month (ongoing retraining)
  • Storage (model versions, training data, logs): $8K/month (16x higher)
  • Monitoring and observability: $12K/month (not budgeted)
  • Data pipeline infrastructure: $15K/month (not budgeted)
  • Redundancy and failover: $20K/month (not budgeted)
  • Total: $125K/month (23x initial estimate)

And that’s just ONE model.

Why @cto_michelle’s 5x Budget Rule is Actually OPTIMISTIC

@cto_michelle said budget 5x what you think. The Stripe speaker said:

“In my experience, production AI costs 10-20x your POC costs. If your finance team won’t accept that multiplier, don’t start the project.”

Why the massive difference?

  1. POC runs on toy data (1000 records)

    • Production runs on real data (100M records)
    • Scale factor: 100,000x

  2. POC has no reliability requirements

    • Production needs 99.99% uptime (4 nines)
    • That means redundancy, failover, chaos testing
    • Cost multiplier: 3-5x

  3. POC doesn’t handle edge cases

    • Production must handle every possible input
    • Cost multiplier: 2-3x (just in defensive coding and error handling)

  4. POC doesn’t retrain models

    • Production models degrade (data drift)
    • Must retrain every 3-6 months
    • Cost multiplier: Ongoing (not just initial training cost)

  5. POC doesn’t monitor performance

    • Production needs real-time monitoring, alerting, debugging
    • Tools cost money. Engineers debugging cost WAY more money.

The Skills Gap from an Engineering Manager Perspective

@security_sam and @cto_michelle mentioned the skills gap. Let me add the engineering hiring reality:

My team’s open roles (I’m actively hiring at SF Tech Week):

  • ML Engineer: 250 applications, 12 qualified, 3 offers made, 0 accepted (all got better offers)
  • MLOps Engineer: 42 applications, 6 qualified, 2 offers made, 1 accepted
  • AI Infrastructure Engineer: 18 applications, 2 qualified, 1 offer made, 0 accepted

The math doesn’t work:

We need to hire 10 engineers to support our AI roadmap.
At current acceptance rates, we need to:

  • Source 2,500 ML Engineer candidates to hire 10
  • Source 420 MLOps candidates to hire 10
  • Source 180 AI Infrastructure candidates to hire 10

That’s 3,100 candidates to fill 30 roles.

And we’re competing with OpenAI, Anthropic, Google, Meta - companies that can pay 2-3x what we can afford.

The “Just Use OpenAI API” Trap

Multiple people at this conference keep saying: “Why build your own AI? Just use OpenAI’s API!”

The Stripe speaker addressed this directly:

When OpenAI API works:

  • Low-stakes use cases (content generation, summarization)
  • Low volume (<100K requests/month)
  • Non-latency-sensitive (users can wait 2-5 seconds)
  • No sensitive data

When OpenAI API fails:

  • High-stakes decisions (financial, medical, legal)
  • High volume (>1M requests/month = $$$)
  • Latency-sensitive (<100ms required)
  • Sensitive data (can’t send to third party)

Real example:

A fintech company tried using OpenAI API for fraud detection:

  • POC: 1,000 transactions/day, $50/month, works great
  • Production: 500,000 transactions/day
  • Projected cost: $25,000/month
  • Actual cost after 1 month: $47,000/month (usage patterns differed from POC)
  • Latency: 200-400ms (unacceptable for real-time fraud, users timing out)
  • Data compliance: Legal team said “absolutely not, we can’t send customer data to OpenAI”

Solution:

  • Spent 6 months building in-house model
  • Cost: $800K development + $15K/month infrastructure
  • Latency: 12ms average
  • Data compliance: Approved

Break-even timeline: 18 months

But that doesn’t count the 6-month delay to market.

The Integration Hell @cto_michelle Mentioned

Let me give you a concrete example from my team:

Project: Integrate AI-powered search into our product

Sounds simple, right? AI startup vendors make it sound easy: “Just call our API!”

Actual integration requirements:

  1. Authentication and authorization

    • Our users have 47 different permission levels
    • AI needs to respect those permissions
    • Can’t just “index everything and let AI search it”
    • Took 2 months to build permission-aware indexing

  2. Data synchronization

    • Our data is in 8 different databases
    • Real-time sync vs batch sync trade-offs
    • Took 3 months to build reliable data pipeline

  3. Error handling

    • What if AI API is down? (It will be)
    • Fallback to regular search? Different UX
    • What if AI returns garbage? How do we detect and handle?
    • Took 1 month to build resilient error handling

  4. Performance

    • AI search is slower than regular search
    • Users expect <200ms response
    • Had to add caching layer, prediction, pre-fetching
    • Took 2 months to optimize performance

  5. Monitoring and debugging

    • When AI search is “wrong”, how do we debug?
    • Traditional search: query logs, index stats, straightforward
    • AI search: model version, embedding space, relevance scoring, vector search, black box
    • Took 1 month to build proper monitoring

  6. Cost management

    • AI search costs 10x more than traditional search
    • Can’t let one user’s expensive query bankrupt us
    • Rate limiting, cost tracking, alerting
    • Took 2 weeks to build cost controls

Total: 9.5 months to “just integrate an API”

POC took 2 weeks.

How This Relates to the 80% Post-POC Problem

@cto_michelle’s stat that 80% of work happens after POC is NOT because engineers are slow.

It’s because POC answers: “Can this work in ideal conditions?”

Production answers: “Can this work in ALL conditions, reliably, securely, cost-effectively, at scale, with monitoring, with error handling, with compliance, with integration into existing systems, and with acceptable user experience?”

POC is 1 question. Production is 50 questions.

The Controversial Solution: Don’t Build AI

The Stripe speaker’s most controversial slide:

“The best AI project is the one you DON’T build.”

He showed a decision tree:

  1. Can you solve this with deterministic code? → Don’t use AI
  2. Can you buy a solution? → Don’t build AI
  3. Can you outsource this to a vendor? → Don’t build AI
  4. Is the ROI clear and measurable? → Maybe build AI
  5. Do you have the skills, budget, and timeline? → Maybe build AI
  6. Is this a strategic differentiator? → Consider building AI

By his tree, only ~5% of AI projects should actually be built.

The other 95%? Use existing tools, buy solutions, or accept that the problem doesn’t need an AI solution.

My Team’s New Policy (as of Today)

After hearing all these panels, I’m implementing a new policy for my engineering team:

Before we start ANY AI project:

  1. :white_check_mark: Prove it works with deterministic code first (if possible)
  2. :white_check_mark: Get sign-off on 10x budget multiplier from finance
  3. :white_check_mark: Get commitment for 12-18 month timeline (no shortcuts)
  4. :white_check_mark: Hire compliance engineer BEFORE ML engineer
  5. :white_check_mark: Build full production infrastructure plan before writing ML code
  6. :white_check_mark: Define success metrics that aren’t “AI accuracy” (actual business metrics)

If we can’t check all 6 boxes, we DON’T START.

Questions for Engineering Leaders

  1. What’s your actual AI cost multiplier? POC vs production, be honest

  2. Has anyone successfully hired an AI Infrastructure Engineer? Where did you find them and what did you pay?

  3. For those using OpenAI API in production - what’s your actual monthly bill and request volume?

  4. What percentage of AI projects have you CANCELLED after POC but before production? Why?

Day 2: I’m attending the MLOps tooling showcase. Hypothesis: There are 100+ MLOps tools and none of them solve the actual problem (integration with legacy systems). Let’s see if I’m wrong.

Sources:

  • SF Tech Week “AI Infrastructure at Scale” session (Day 1)
  • Stripe ML Infrastructure team presentation
  • Real project cost and timeline data from my team and the panel
  • Engineering hiring data from my active recruiting efforts this week