2 posts tagged with "staging"

You can stand up a staging database. You can stand up a staging queue, a staging payments sandbox, a staging copy of every third-party API you depend on. For thirty years the whole discipline of pre-production has been built on one assumption: that you can create a faithful-enough replica of production, poke it, and learn something true about what will happen when you ship.

Then you added a hosted model to your critical path, and the assumption quietly broke. The one component whose behavior now dominates your product — the thing that decides what your app actually says and does — is the one component you cannot stand up a staging copy of. It is versioned by someone else, rate-limited by someone else, and quietly updated by someone else on a schedule you don't see. Your staging environment has a staging everything, except a staging model.

A team I know has a textbook staging setup. Read-only replicas of the production database. A mock Stripe account that pretends to charge cards. Synthetic users with fake email addresses on a domain nobody owns. The agent is asked to walk through an "account delinquent" escalation flow in staging, end to end, as part of a release rehearsal. The trace looks clean. The agent does what it is supposed to do.

Three minutes later, a real customer — a paying one, who churned six months ago and was still in a dormant export the developer had used to seed a test fixture — replies to a politely-worded payment-overdue email. The "send_email" tool, registered next to a dozen other tools that all terminate in mocks, was wired to the production Mailgun key. The developer who set it up two sprints earlier had been iterating fast on email templates and the sandbox tier capped them at five emails an hour, which broke the inner loop, so they swapped in the real key "just for the afternoon" and forgot. Nobody re-checked. The agent had no way to know.