I need to get something off my chest: my team spent 18 months building out a Kubernetes infrastructure for our legacy banking system modernization project. We’re now seriously evaluating whether we should have just used a modern PaaS instead.
This feels like admitting failure, but I’m increasingly convinced we over-engineered our solution.
The Kubernetes Reality
Here’s what we learned the hard way: 80% of our production incidents are from operational complexity, not infrastructure failures.
- Certificate rotation broke because someone misconfigured cert-manager
- A namespace resource quota caused mysterious pod evictions
- Helm chart version incompatibilities took down staging for 6 hours
- Our service mesh added 200ms of latency we spent weeks debugging
None of these are Kubernetes’ fault. They’re the inevitable complexity of running a sophisticated orchestration platform. But the question I keep asking: did we need that sophistication?
The PaaS Evolution
When we made the K8s decision back in 2024, PaaS meant Heroku or limited platforms that couldn’t handle our scale or compliance needs. But the landscape has changed dramatically:
- Railway, Render, Fly.io have evolved with serious enterprise features
- They handle databases, CDNs, background jobs, cron - the full stack
- Regional deployment, auto-scaling, zero-downtime deploys are table stakes
- Some even offer compliance certifications we need for financial services
The Cost That Nobody Talks About
We have 3 full-time DevOps engineers whose primary job is maintaining our Kubernetes infrastructure. That’s not including the on-call rotation burden on the entire engineering team.
Let’s do the math:
- 3 DevOps engineers × K average = K/year
- Opportunity cost: what could those engineers build instead?
- Cognitive load on product engineers who have to understand K8s to deploy
Compare that to managed PaaS pricing. Even at K-3K/month for our scale, we’re talking -36K/year. The ROI seems obvious when you frame it that way.
The Question I’m Wrestling With
At what scale does Kubernetes complexity actually pay off?
We’re not Netflix. We’re not running 10,000 microservices. We have maybe 30 services supporting a few million transactions per day. Could a modern PaaS handle that? Probably.
The Regulatory Compliance Angle
Here’s where I need help from this community: we’re in financial services. We have audit requirements, compliance frameworks, data sovereignty concerns.
My CFO loves the idea of PaaS (lower cost, predictable pricing). My CISO is nervous (less control, vendor dependency for compliance). My engineering team is split (platform team wants to keep K8s, product engineers want simpler deploys).
Questions for the group:
- Has anyone successfully run regulated workloads (fintech, healthcare, etc.) on modern PaaS platforms?
- At what team size or transaction volume did PaaS limitations actually force you to Kubernetes?
- If you’re running K8s, what percentage of your engineering time goes to platform maintenance vs product features?
- Are there hybrid approaches that make sense? (PaaS for most services, K8s for specific needs?)
I’m genuinely open to the possibility that K8s was the right choice and I’m just experiencing temporary frustration. But I’m also open to the possibility that we cargo-culted a solution that made sense for Google-scale problems but not for our reality.
What am I missing?