FinOps in 2026: AI Costs Are Eating Everything - How Do We Even Budget This?

General
1

I’m preparing our Series C fundraising materials, and investors keep asking the same question: “What’s your AI infrastructure cost as a percentage of revenue?” I genuinely don’t have a good answer, and it’s making me question whether we understand our own unit economics.

AI Cost Management is Now Priority #1

According to the latest State of FinOps 2026 report, AI cost management is the #1 desired skillset across organizations of all sizes - 98% adoption, up from just 63% last year. That’s a massive shift.

And I understand why. AI costs are eating our cloud budget.

Our cloud spend breakdown (2024 vs 2026):

2024:

  • Compute: K/month (65%)
  • Storage: K/month (17%)
  • Data transfer: K/month (12%)
  • Other: K/month (6%)
  • Total: K/month

2026:

  • Traditional compute: K/month (29%)
  • AI/ML compute (GPUs): K/month (52%)
  • Storage: K/month (11%)
  • Data transfer: K/month (7%)
  • Other: K/month (1%)
  • Total: K/month

AI infrastructure went from 0% to 52% of our cloud spend in 18 months. And it’s wildly unpredictable.

The Forecasting Nightmare

Traditional infrastructure is predictable:

  • “We need 10 servers to handle 100K users”
  • “Each server costs /month”
  • “If we grow to 150K users, we need 15 servers”
  • Math works out, forecasting is straightforward

AI infrastructure is chaos:

  • GPU spot instance prices swing 10x based on availability
  • Model inference costs depend on request complexity (token count, image size)
  • Training runs have unpredictable duration (convergence isn’t linear)
  • New models launch monthly with different cost/performance tradeoffs

Example from last month:

We budgeted K for GPU compute. Actual spend: K.

Why?

  • Spot instance availability dropped mid-month (AWS announced new AI services)
  • Fell back to on-demand instances at 3x cost
  • Product launched a new feature with longer AI prompts (more tokens = higher cost)
  • Engineering experimented with larger model (needed more GPU memory)

How do I explain this variance to our CFO? To potential Series C investors?

FinOps Is Now an Engineering Problem

The State of FinOps report shows 78% of FinOps teams now report to CTO/CIO, up from 60% in 2023. This isn’t a finance function anymore - it’s a technical architecture decision.

And that makes sense. The biggest cost optimization opportunities aren’t in vendor negotiation - they’re in:

  • Which AI model to use (GPT-4 vs Claude vs open-source)
  • Model quantization and optimization techniques
  • Batch processing vs real-time inference
  • GPU instance type selection and scheduling
  • Prompt engineering to reduce token usage

These are engineering decisions with massive financial implications. But my engineering team doesn’t think about costs, and my finance team doesn’t understand AI architecture.

FinOps Expanding Beyond Cloud

Here’s what’s making my job even harder: FinOps is no longer just cloud costs.

According to the 2026 report:

  • 90% managing SaaS costs (up from 65%)
  • 64% managing software licensing (up 15%)
  • 57% managing private cloud (up 18%)
  • 48% managing data centers (up 12%)

We’re supposed to be “cloud finance” experts, but now I’m tracking:

  • AWS/GCP cloud spend
  • OpenAI API costs
  • Anthropic API costs
  • Hugging Face inference endpoints
  • GitHub Copilot seats
  • Datadog subscription
  • SaaS tools across the company
  • Software licenses for development tools

I need a unified view of all technology spending, but every vendor has different billing models, different APIs, different reporting.

The Questions I’m Struggling With

1. How do I budget AI costs when they’re fundamentally unpredictable?

Should I:

  • Use historical average + 50% buffer? (Feels arbitrary)
  • Model per-request costs and multiply by traffic forecast? (Traffic forecasts are also wrong)
  • Just tell the board “AI costs will fluctuate 30-40%”? (They won’t love this)

2. How do I align engineering incentives with cost optimization?

Engineers want the best model for the best user experience. Finance wants predictable, optimized costs. These often conflict.

Do I:

  • Make cost metrics part of engineering performance reviews? (Feels wrong)
  • Create cost budgets per team? (Creates friction and gaming)
  • Just accept that this is a business cost of AI-driven products? (CFO says no)

3. What FinOps tools actually work for AI costs?

We’ve tried:

  • AWS Cost Explorer (useless for AI-specific analysis)
  • Custom dashboards (maintenance burden)
  • Third-party FinOps tools (expensive, don’t handle AI well)

Are there AI-specific cost management tools that actually deliver value? Or is this still too nascent?

4. How do other companies model AI cost in unit economics?

For our Series C pitch, investors want to see:

  • Cost per user
  • Gross margin
  • Path to profitability

But AI costs break traditional unit economics:

  • One power user might cost 100x more than average user (based on AI usage)
  • We can’t easily attribute AI costs to specific customers
  • Costs depend on model choice, which changes quarterly

How are other AI-driven companies modeling this for investors?

Looking for Perspectives

I’d love to hear from:

  • Engineering leaders: How do you think about AI costs in architecture decisions? Do you have cost budgets? How do you balance performance vs cost?

  • Other finance folks: How are you forecasting AI costs? What models or frameworks work? How do you present this to boards and investors?

  • FinOps practitioners: What tools and practices actually help with AI cost management?

This feels like a 2026-specific problem where best practices are still being written. Help me learn from your experiences.

2

Carlos, this is the conversation Finance and Engineering need to have more often. I’m glad you’re raising this because the traditional separation between “engineering builds it” and “finance pays for it” doesn’t work in the AI era.

How We’re Tackling This

At my company, we made a deliberate organizational change 6 months ago: FinOps is now in engineering leadership meetings, not just finance reviews.

Here’s why: The biggest cost optimization opportunities aren’t in vendor negotiation or budget reviews. They’re in daily engineering decisions that have 10x cost implications.

Real Example: Model Choice Drives Economics

Last quarter, we faced a decision on our recommendation engine:

Option A: GPT-4

  • Best quality recommendations
  • Cost: ~/bin/zsh.12 per user session
  • Projected monthly cost at 500K users: K

Option B: Claude 3.5 Sonnet

  • 95% quality of GPT-4 for our use case
  • Cost: ~/bin/zsh.06 per user session
  • Projected monthly cost: K

Option C: Fine-tuned Llama 3.3 (self-hosted)

  • 85% quality, acceptable for most users
  • Infrastructure cost: K/month (GPUs)
  • Marginal cost per user: ~/bin/zsh.005
  • Projected monthly cost: .5K total

This is fundamentally an architecture decision with massive financial implications. Engineering wanted Option A (best quality). Finance wanted Option C (lowest cost). Product wanted Option B (balanced).

We chose Option B for launch, with a roadmap to Option C.

Rationale:

  • Time to market matters (fine-tuning takes 2-3 months)
  • Quality threshold meets user needs (validated in testing)
  • K/month savings vs Option A funds 2 additional engineers
  • Path to further optimization exists when we have more data

Aligning Engineering Incentives

You asked: “How do I align engineering incentives with cost optimization?”

Here’s what we did:

1. Make costs visible
Every service now has a cost-per-request dashboard. Engineers see the financial impact of their code in real-time, not in monthly finance reviews.

We use custom telemetry + Weights & Biases + cloud billing APIs to show:

  • Inference cost per request
  • Training cost per model iteration
  • GPU utilization and waste
  • Cost trends over time

2. Cost as an engineering metric, not a constraint
We don’t say “you have a K budget.” We say “here’s the cost, here’s the business value, optimize the ratio.”

Engineers optimize for cost-per-value, not absolute cost. This encourages smart optimization, not just cheapness.

3. Cost optimization is celebrated, not mandated
When an engineer reduces inference cost through prompt optimization or model quantization, we recognize it in team meetings. It’s engineering excellence, not penny-pinching.

Cross-Functional Collaboration

The biggest shift: Engineering, Product, and Finance review AI costs together monthly.

Agenda:

  • Finance shows cost trends and variances
  • Engineering explains technical drivers (model changes, usage patterns)
  • Product shares feature roadmap and expected cost impact
  • We make joint decisions about optimization priorities

This sounds bureaucratic, but it’s actually streamlined decision-making. We’re not doing separate engineering reviews and finance reviews - we’re looking at the same data with different expertise.

To Your Specific Questions

How do I budget AI costs when they’re fundamentally unpredictable?

We use scenario-based budgeting:

  • Base case: Current model, current traffic growth
  • Optimization case: Planned model improvements
  • Growth case: Higher traffic without optimization
  • New feature case: Known product roadmap additions

We give the CFO ranges, not single numbers. “AI costs will be K-65K/month depending on traffic and optimization success.”

Investors actually appreciate this more than false precision. It shows we understand the variables.

What FinOps tools actually work for AI costs?

Honestly, we built custom dashboards. Off-the-shelf tools weren’t great for AI-specific metrics.

But the tool matters less than the practice. Having Engineering and Finance look at the same dashboard together is more valuable than any fancy FinOps platform.

The Question I’ll Ask You

Carlos, you showed AI went from 0% to 52% of cloud spend. What’s the revenue impact of that AI investment?

If AI features drove 3x user engagement or 2x conversion rate, then 52% of infrastructure cost might be the best money you’re spending.

If AI features are table stakes and don’t move metrics, then you have a product strategy problem, not just a cost problem.

Finance and Engineering need to answer that together. What does your data show?

3

Carlos and Michelle, this is hitting exactly what I’m dealing with on the infrastructure side. Let me share what we’re doing for AI cost optimization at the technical level.

Cost Optimization is Daily Work, Not Quarterly Review

Michelle’s point about real-time visibility is critical. We’ve made cost optimization part of our daily engineering practice, not something finance reviews after the fact.

Specific Technical Optimizations

1. Model Quantization

We reduced our primary model from FP32 to INT8 quantization:

  • Model size: 1.2GB → 350MB (70% reduction)
  • Inference latency: 180ms → 95ms (47% faster)
  • GPU memory: 4GB → 1.2GB (can fit 3x more on same instance)
  • Quality degradation: ~3% on our metrics (acceptable)
  • Cost impact: K/month → K/month

This required 2 weeks of engineering time to validate quality and optimize. ROI was immediate.

2. Batch Processing for Non-Real-Time Workloads

We were running inference on-demand for everything. We split workloads:

  • Real-time user requests: Immediate inference (can’t batch)
  • Analytics and reporting: Batched hourly
  • Content recommendations: Pre-computed daily

Batching benefits:

  • GPU utilization: 35% → 78%
  • Cost per inference: /bin/zsh.08 → /bin/zsh.03
  • Savings: K/month on analytics workload

3. Spot Instance Strategy

Carlos, you mentioned spot instance price volatility. We’ve automated our spot strategy:

  • Monitor spot prices across instance types and regions
  • Automatically migrate to cheapest available spot instance
  • Fall back to on-demand only when spot unavailable
  • Use multiple instance types (not locked to specific GPU)

Results:

  • Average discount vs on-demand: 68%
  • Spot interruption rate: ~8% of instances
  • Automated recovery: <2 minutes
  • Savings: K/month

This required building custom orchestration, but at our scale it pays for itself monthly.

4. Prompt Engineering as Cost Optimization

For our API that uses GPT-4, we optimized prompts:

  • Reduced system prompt tokens: 850 → 320
  • Structured outputs to reduce response tokens
  • Cached common instructions in application layer
  • Use function calling instead of verbose prompts

Impact:

  • Average tokens per request: 2,400 → 1,100
  • Cost per request: /bin/zsh.15 → /bin/zsh.07
  • Monthly savings: K (at 100K requests/month)

Engineers now think about token efficiency the same way they think about database query optimization.

Cost-Per-Request Dashboard

We built a dashboard that shows every engineer:

  • Cost per API endpoint
  • Cost trends over the last 30 days
  • Breakdown: model inference vs infrastructure vs data transfer
  • P50/P95/P99 cost percentiles
  • Cost anomalies and spikes

This makes cost visible and actionable. When an engineer sees their endpoint costs spike 3x, they investigate immediately, not after finance sends an alert weeks later.

Technical Stack for Cost Monitoring

  • Telemetry: Custom middleware that logs model usage, tokens, latency
  • Storage: ClickHouse for fast aggregation of billions of inference events
  • Visualization: Grafana dashboards with cost metrics
  • Alerting: PagerDuty when cost anomalies detected
  • Billing API integration: Pull cloud billing data hourly, correlate with usage

To Michelle’s Model Choice Example

I want to add a fourth option you didn’t mention:

Option D: Hybrid routing

  • 80% of requests: Fine-tuned Llama (cheap, fast, good enough)
  • 15% of requests: Claude 3.5 Sonnet (higher quality needed)
  • 5% of requests: GPT-4 (highest quality for edge cases)

Route based on request characteristics:

  • Simple queries → Llama
  • Medium complexity → Claude
  • Complex or high-value → GPT-4

Potential cost:

  • Llama: 400K requests × /bin/zsh.005 = K
  • Claude: 75K requests × /bin/zsh.06 = .5K
  • GPT-4: 25K requests × /bin/zsh.12 = K
  • Infrastructure: K
  • Total: .5K vs K for pure GPT-4

This requires request classification logic and routing infrastructure, but the cost savings can be massive.

The Cultural Shift

Carlos asked: “How do I align engineering incentives with cost optimization?”

Here’s what worked for us: Make cost visibility automatic, not mandated.

We don’t have cost budgets per team (creates gaming and finger-pointing). We just make the costs visible and let engineers do what they do best: optimize.

When engineers see:

  • “This endpoint costs /bin/zsh.50 per request”
  • “We’re processing 10K requests/day”
  • “That’s K/month just for this feature”

They naturally ask: “Can we make this cheaper?” Not because finance told them to, but because engineers like solving optimization problems.

To Your Question About Tools

What FinOps tools actually work for AI costs?

Honestly, we built custom tooling because:

  1. AI costs span multiple vendors (AWS, OpenAI, Anthropic, Hugging Face)
  2. Off-the-shelf tools don’t correlate usage with business metrics
  3. We need real-time visibility, not monthly reports

Our stack:

  • Custom telemetry for AI-specific metrics
  • ClickHouse for cost data warehouse
  • Grafana for dashboards
  • Python scripts for anomaly detection
  • Slack alerts for cost spikes

Not fancy, but effective.

The Question About Unit Economics

How do you model AI cost in unit economics?

We track:

  • Cost per active user per month (blended AI costs)
  • Cost per API request (granular view)
  • Cost per value metric (e.g., cost per recommendation clicked)

For investors, we show:

  • Current AI cost per user: .40/month
  • Target after optimizations: /bin/zsh.80/month
  • Path to target: Specific engineering initiatives (quantization, batching, model switching)

Investors want to see that you understand the cost structure and have a plan to improve it. Specific numbers with specific optimization plans are more credible than “we’ll figure it out.”

4

Carlos, this is a conversation we should have had months ago. Finance and Engineering need to speak the same language, especially around AI costs. Let me share how we’re approaching this in financial services.

The Organizational Structure Change

Six months ago, we created a “FinOps Engineer” role that reports to both me (Director of Engineering) and our VP Finance. This person bridges the gap.

Responsibilities:

  • Translate cloud billing into engineering metrics
  • Build cost visibility dashboards
  • Partner with platform team on cost optimization
  • Present to finance team in their language (unit economics, forecasting)
  • Present to engineering team in our language (latency, throughput, optimization)

This role has been transformational. Before, finance would send us a bill and ask “why did costs increase?” We’d shrug and say “more users?” Now we have detailed attribution.

Cost Attribution at the Service Level

We tag every cloud resource with:

  • Team ownership
  • Service name
  • Environment (prod, staging, dev)
  • Cost center
  • Product area

This lets us answer:

  • “What does the mortgage processing service cost per transaction?”
  • “How much does Team A’s infrastructure cost vs Team B?”
  • “What’s the cost trend for our new AI features?”

Real example:

Our fraud detection AI feature:

  • Infrastructure cost: K/month
  • Processed transactions: 850K/month
  • Cost per transaction: /bin/zsh.0165
  • Revenue per transaction (interchange): /bin/zsh.12
  • Margin contribution: /bin/zsh.1035 per transaction

When we show finance that AI fraud detection has 86% margin, they stop asking us to cut costs and start asking us to scale it.

The Forecasting Model We Use

Carlos, you asked how to forecast AI costs. Here’s our framework:

1. Decompose into components

Don’t forecast “AI costs” as one number. Forecast:

  • Training costs (mostly one-time or periodic)
  • Inference costs (scales with usage)
  • Infrastructure baseline (GPUs, storage)
  • API costs (third-party models)

2. Model cost drivers, not total costs

For inference costs:

  • Cost per 1K transactions
  • Expected transaction volume
  • Growth rate assumptions
  • Optimization roadmap impact

3. Build scenarios

  • Baseline: Current state, current efficiency
  • Optimistic: Growth + planned optimizations
  • Pessimistic: Growth without optimizations, spot price increases

We present all three to finance. Board sees ranges, not false precision.

Example Q2 2026 forecast:

  • Baseline: K/month
  • Optimistic: K/month (includes quantization project)
  • Pessimistic: K/month (includes new feature without optimization)

Our actual spend: K/month (within baseline range)

Team Training: Engineers Learn FinOps

We mandated that every engineer on our platform team complete:

  1. Cloud economics training (2-day workshop)

    • How cloud billing works
    • Common cost anti-patterns
    • ROI calculation for infrastructure decisions

  2. Cost visibility tools training (1-day hands-on)

    • How to use our cost dashboards
    • How to investigate cost spikes
    • How to estimate cost of new features

Impact:

Engineers now proactively ask:

  • “What’s the cost of this design choice?”
  • “Should we use managed service X or build it ourselves?”
  • “Is this optimization worth the engineering time?”

These are questions finance can’t answer alone. Engineers need to own cost implications of technical decisions.

Cross-Functional Meetings That Work

We do monthly “Cost & Value” reviews:

Attendees:

  • Engineering (me + platform lead)
  • Finance (VP Finance + FinOps analyst)
  • Product (product managers for cost-heavy features)

Agenda:

  1. Finance presents: Cost trends, variances, forecasts
  2. Engineering presents: Technical drivers, optimization opportunities
  3. Product presents: Feature roadmap and expected cost impact
  4. Joint discussion: Priorities for next month

Key rule: No blame, only understanding.

When costs spike, we don’t ask “whose fault?” We ask “what changed and how do we optimize?”

To Your Investor Question

Carlos, you asked: “How do other companies model AI cost in unit economics for investors?”

For our fintech:

Metrics we show investors:

  1. Cost per transaction (including AI fraud detection)

    • Current: /bin/zsh.084
    • Target (end of year): /bin/zsh.052
    • Path to target: Specific optimizations with timelines

  2. AI cost as % of revenue

    • Current: 18%
    • Target: 12%
    • Industry benchmark: 15-20% for AI-driven fintech

  3. Gross margin trend

    • Show that AI costs are declining per transaction as we scale
    • Demonstrate unit economics improve with volume

What investors want to see:

  • You understand the cost structure
  • You have a plan to optimize
  • Costs improve with scale (operating leverage)
  • You’re not just throwing money at AI without measuring ROI

The Hard Question

Michelle asked you: “What’s the revenue impact of that AI investment?”

This is the question finance and engineering must answer together.

For us:

  • AI fraud detection prevents .4M/year in fraud losses
  • Infrastructure cost: K/year
  • ROI: 14x

But we only know this because we measure both costs (engineering) and impact (finance + product).

Carlos, do you have similar analysis for your AI features? If AI is 52% of infrastructure but drives 80% of user value, that’s a great story. If it’s 52% of cost for 10% of value, you have a strategic problem to address with product and engineering leadership.