Ephemeral Preview Environments Changed Our PR Review Process — Here's Our Vercel-Like Setup on Kubernetes

General
1

If your PR review process looks anything like ours did six months ago, here’s the honest picture: a developer opens a pull request with frontend and backend changes, a reviewer glances at the diff, maybe leaves a comment about a variable name, types “LGTM,” and approves. Nobody actually runs the code. Nobody tests the user-facing behavior. Visual bugs, broken interactions, and integration issues sail through review and land in staging — or worse, production.

The root problem is friction. To properly review a full-stack PR, the reviewer has to stash their current work, pull the branch, install dependencies, seed the database, run both the API server and frontend, and then navigate to the right page to test the change. That’s 10-15 minutes of setup for a 5-minute review. So rational people skip it and just read the code.

How We Built Ephemeral Preview Environments

We built a system where every pull request automatically gets a full-stack deployment with a unique URL: pr-123.preview.company.dev. The reviewer clicks the link, sees the running application with the PR’s changes, and can actually interact with it. No local setup required.

Here’s the architecture:

Namespace-per-PR on Kubernetes. Each PR gets its own Kubernetes namespace (preview-pr-123) containing the full application stack. This gives us resource isolation and clean teardown — deleting the namespace removes everything.

GitOps with ArgoCD. When a PR is opened, a GitHub Action generates Helm values with the PR number templated in (image tags, ingress hostnames, namespace name) and commits them to our GitOps repo. ArgoCD detects the change and deploys the full stack. This means our preview deployments go through the exact same deployment pipeline as staging and production.

Database seeding. Each preview environment gets its own PostgreSQL instance seeded with a synthetic test dataset. We built a seeding tool that generates realistic but fake data covering our main user flows — accounts, transactions, settings, the works. Seed time is about 45 seconds for our schema.

Automatic TLS and DNS. cert-manager handles Let’s Encrypt certificates for each preview URL via DNS-01 challenges. ExternalDNS creates wildcard DNS entries pointing *.preview.company.dev to our ingress controller. Zero manual configuration per PR.

Scale-to-zero. This was critical for cost management. After 30 minutes of inactivity (no HTTP requests), KEDA scales the deployments to zero replicas. The first request after scaling triggers a cold start of about 15 seconds — not instant, but acceptable for review purposes. When the PR is closed or merged, a GitHub Action deletes the entire namespace.

The Numbers

  • Average spin-up time: 3 minutes from PR open to live URL
  • Monthly cost: ~$200 for approximately 50 active preview environments (thanks to aggressive scale-to-zero)
  • Infrastructure: Single EKS cluster with spot instances for preview workloads

Impact on Review Quality

The change in reviewer behavior was immediate and dramatic. Reviewers now actually click around the application, test user flows, and catch issues that are invisible in a code diff. Comments shifted from “this variable name could be better” to “the loading spinner on the settings page persists even after data loads” and “the error state is cut off on mobile viewports.”

PR approval time decreased by roughly 30% because reviewers feel more confident approving when they’ve actually seen the change running. Counterintuitively, the number of review comments increased — but they were higher quality, catching real bugs instead of bikeshedding code style.

Challenges We’re Still Solving

Database schema complexity. Our seeding tool needs to be updated every time we run a migration. We’re exploring using a recent anonymized snapshot approach instead, but that introduces its own concerns.

Secrets management. Preview environments need API keys for third-party services (Stripe test mode, SendGrid, etc.). We use Sealed Secrets with a separate set of credentials from production, but managing the rotation across ephemeral namespaces is painful.

Config drift. Occasionally something works perfectly in preview but breaks in staging due to environment configuration differences. We’ve reduced this by sharing Helm value templates between preview and staging, but it’s not fully eliminated.

Alternatives Considered

If you’re running a static site or simple frontend, just use Vercel or Netlify — they do preview deployments out of the box and it’s not worth building custom infrastructure. Railway and Render are adding preview environment features for full-stack apps, and they’re getting better rapidly.

But for complex microservice architectures with multiple databases and service dependencies, we found that no managed platform could spin up our full topology without significant customization. The custom K8s approach gave us the flexibility we needed.

The question for the community: Is the investment in custom preview infrastructure worth it for your team, or are managed platforms sufficient? Where’s the complexity threshold where build-your-own becomes the right call?

2

We went through this exact evaluation last quarter, and your write-up mirrors our experience almost point for point. The build-vs-buy decision ultimately came down to our stack complexity.

We run 6 microservices, 2 databases (Postgres and Redis), and a RabbitMQ message queue. When we evaluated managed platforms — Railway, Render, even some of the newer preview-environment-as-a-service startups — none of them could spin up that full topology per PR without significant glue code and workarounds. At that point, you’re maintaining custom infrastructure anyway, just with less control over the underlying platform.

We ended up with a very similar K8s-based approach but made a different orchestration choice: Argo Workflows instead of GitHub Actions. The reasoning was practical — Argo Workflows gives us better retry logic for flaky steps (database seeding occasionally fails on resource contention), parallel step execution (we spin up all 6 services simultaneously rather than sequentially), and the workflow definitions live alongside our Kubernetes manifests in the GitOps repo. It keeps the entire preview environment lifecycle within the Kubernetes ecosystem rather than split between GitHub and K8s.

The ROI was unmistakably clear within 2 months. We caught 3 major UX regressions in preview environments that absolutely would have shipped to production:

  1. A checkout flow that silently failed when the cart had more than 50 items — the API returned a paginated response that the frontend didn’t handle.
  2. A timezone rendering bug that only manifested with non-US locale settings, which none of our US-based developers would have caught locally.
  3. A race condition in our notification service that caused duplicate emails — only visible when testing the full service mesh, not in unit or integration tests.

Each of those would have been a production incident. The preview infrastructure paid for itself on the first catch alone.

My advice to teams evaluating this: if you’re a monolith or a simple frontend app, use Vercel or Netlify preview deployments. Full stop. Don’t build custom infrastructure you don’t need. But if you’re running microservices with inter-service dependencies, invest in the custom K8s setup. The upfront cost is real — we spent about 3 engineering weeks building it — but the ongoing value in review quality and bug prevention is substantial.

One thing I’d add to Alex’s setup: we run a lightweight smoke test suite against each preview environment automatically. It hits the main user flows via Playwright and posts the results as a PR comment. Reviewers get a green checkmark before they even click the preview link, which filters out the obviously broken PRs and saves reviewer time.

3

The review quality improvement Alex describes resonates deeply with my experience managing engineering teams through this exact transition.

We piloted preview environments for our largest team (12 engineers, full-stack product team) and the data was striking: visual and interaction bugs caught during code review increased by 340% in the first quarter after adoption. That’s not a typo. The baseline was embarrassingly low — before preview environments, reviewers were catching almost no visual bugs because they simply weren’t running the code.

But there’s a cultural component here that’s just as important as the technical infrastructure. When reviewers can see the change running in a real browser, they give fundamentally different feedback. The shift I observed was from code-level feedback to product-level feedback.

Before preview environments, review comments looked like:

  • “Consider extracting this into a utility function”
  • “This query might be slow on large datasets”
  • “Nit: rename this variable”

After preview environments, we started seeing:

  • “The loading state on the settings page feels janky — the skeleton flickers for about 200ms before content appears”
  • “The error message is cut off on mobile when the username is longer than 20 characters”
  • “This workflow feels like it needs a confirmation step before the destructive action”

That second category of feedback is enormously more valuable. It’s the kind of thing that typically only gets caught in QA or, worse, by users in production. Preview environments essentially shifted product-quality feedback left into the PR review stage.

The one thing I’d add to Alex’s setup that made a big difference for us: we require at least one screenshot or screen recording in the PR description showing the reviewer what to test in the preview environment. The PR author includes a brief testing guide — “Navigate to Settings > Notifications, toggle the email preference, verify the confirmation toast appears.”

Without that guidance, we found that reviewers would click the preview link, land on the homepage, not know where the change was, and default back to just reading the diff. The testing guide focuses the reviewer’s attention and makes the preview environment actually useful rather than just available.

We also track a metric we call “preview engagement rate” — the percentage of PRs where the reviewer actually visits the preview URL (we can see this in our access logs). It started at about 40% and climbed to 85% after we introduced the mandatory testing guide. The remaining 15% are typically backend-only PRs where the preview adds less value.

4

Please tell me you’re not seeding preview databases with production data.

I’ve audited companies where preview environments became a massive security hole, and the pattern is always the same: developers clone production databases “for realistic testing” and suddenly every PR reviewer has access to real customer data via a publicly accessible URL. In one case, the preview URLs weren’t even behind authentication — anyone with the URL pattern could enumerate PR numbers and access environments containing PII.

Alex, your setup sounds well-architected, but I want to flag the security considerations that are easy to overlook when the focus is on developer productivity:

1. Synthetic data generation is non-negotiable. Your database seeding must use generated fake data, not anonymized production data (anonymization is notoriously leaky — Netflix famously demonstrated re-identification attacks on “anonymized” datasets). Build a proper test data factory that generates realistic but entirely synthetic records. Yes, this is additional engineering work. Yes, it’s worth it.

2. Authentication on preview URLs. Even basic HTTP authentication is better than nothing. Your preview environments are running on publicly resolvable DNS entries with valid TLS certificates. Without authentication, they’re functionally public websites running your application with known test credentials. We’ve seen attackers use preview environments as reconnaissance tools — they reveal API structures, error handling behavior, and internal tooling that isn’t visible on production.

3. Secrets rotation separate from production. This is the one I see violated most often. Preview environments should use entirely separate API keys, database credentials, and third-party service accounts from staging and production. If a preview environment is compromised (and the attack surface is large — every PR creates a new one), the blast radius should be contained to test accounts. I’ve audited companies where preview environments had the same Stripe API keys as staging. One misconfiguration away from charging real customers from a PR branch.

4. Network policies preventing cross-namespace access. Your Kubernetes namespace isolation is good, but are you enforcing NetworkPolicies that prevent preview namespaces from reaching production or staging services? Default Kubernetes networking allows cross-namespace traffic. A compromised or buggy preview deployment could potentially reach production databases if network policies aren’t explicit.

5. Egress controls. Preview environments running arbitrary PR code should have restricted egress. A malicious or compromised PR could exfiltrate data, mine cryptocurrency, or attack external systems from your infrastructure. At minimum, restrict egress to known-good endpoints.

The convenience of ephemeral environments doesn’t excuse security shortcuts. Every preview environment is a temporary expansion of your attack surface. Treat them accordingly.

I’d be happy to share our preview environment security checklist if there’s interest — we developed it after a penetration test found 4 critical issues in a client’s preview setup.