The World Has No Staging Environment

Your own stack has three environments. Dev is disposable, staging is production-shaped, and prod is sacred. Twenty years of engineering culture — CI gates, canary deploys, blue-green rollouts — all rest on that tiering. Then you give an agent a tool that calls Salesforce, QuickBooks, or Gmail, and the tiering silently evaporates. There is no staging Salesforce for your customer's org. There is no shadow copy of the invoice ledger. The moment a tool call crosses your network boundary into a third-party SaaS, there is exactly one environment, and it is production.

This is the least discussed gap in agent engineering. We have gotten good at sandboxing the agent's compute — containers, egress allowlists, resource caps. We have gotten passable at evaluating the agent's reasoning — offline evals, LLM-as-judge, trajectory scoring. But the agent's actions on the outside world still run against live systems holding real customer data, because for most SaaS surfaces nothing else exists. Teams quietly resolve this the only two ways they can: test against production, or don't test at all.

Why Your Dry-Run Flag Doesn't Count

The first instinct is to build the safety tier yourself: add a dry_run parameter to every write tool, log the payload instead of sending it, and call the system tested. This is better than nothing, and it is also mostly theater.

A dry run on your side of the API boundary validates that your agent composed a request. It cannot tell you what the vendor would have done with it. The failure modes that actually hurt live on the far side of the wire:

• Vendor-side validation. Salesforce will reject an opportunity update because of a validation rule an admin added last quarter. Your dry run doesn't know that rule exists. The agent's plan looked flawless in the log and dies on step three in production.
• Side effects you can't see from the request. Creating an invoice in QuickBooks may trigger an email to the customer. Updating a CRM stage may fire ten downstream automations. The request payload contains zero information about what it detonates.
• Rate limits and quotas. An agent that retries enthusiastically will discover the vendor's rate limiter in production, at the exact moment it is doing something important, and possibly get your whole integration key throttled.
• State-dependent behavior. The same API call succeeds or fails depending on the record's current state — a closed accounting period, a locked opportunity, a thread that was archived between read and write. Dry runs execute against no state at all.

There's a deeper problem than any single failure mode: dry runs break the agent loop itself. Agents are feedback machines — they read the result of one action to decide the next. A dry-run tool returns a synthetic "ok," which means every step after the first is conditioned on a fiction. You aren't testing the trajectory your agent will take in production; you're testing a trajectory that can't occur.

The Ladder of Substitutes, and Where Each One Breaks

If dry runs are theater, what do teams actually use? In practice there's a ladder, and each rung trades fidelity against safety.

Mocks and hand-written fakes sit at the bottom. They're fast, free, and encode your team's assumptions about the vendor's behavior — which is precisely the thing you needed to test. A mock never surprises you, and surprise is the failure mode.

Record-replay proxies are the workhorse middle rung. You put a proxy at the tool boundary, record real request-response traffic once, and replay it deterministically in CI — the VCR pattern, now appearing in agent-specific forms that speak MCP's JSON-RPC directly. This is genuinely useful: it makes regression tests cheap, deterministic, and safe. But replay only covers trajectories you've already seen. The first time the agent takes a novel path — which is the defining behavior of an agent — there's no cassette, and you're back to live traffic.

Replay also rots. Vendors change response shapes, and a recorded fixture from March can pass tests for an API that stopped behaving that way in May.

Simulated environments are what the research community reached for. τ-bench, the benchmark that exposed how unreliable tool-using agents are in customer-service domains, works by simulating both the user and the backing database, then diffing final database state against a goal state. Even frontier models passed fewer than half the airline tasks, and reliability collapsed further when the same task was run eight times. That result should be read as an argument about environments, not just models: the only reason we know those agents are inconsistent is that someone built a high-fidelity simulator where running the same scenario eight times was free. You cannot measure pass^8 against your customer's live Salesforce org.

Vendor sandboxes are the top rung — the vendor's own code, run against disposable state. When they exist and are faithful, nothing else comes close. Which brings us to the uncomfortable part.

The Vendor Sandbox Lottery

The quality of your agent's rehearsal tier is currently determined by which SaaS products your customers happen to use. It's a lottery.

Stripe is the existence proof that this can be done well. Sandboxes are fully isolated environments with their own API keys, webhooks, and data — your CI, staging, and each developer can point at separate sandboxes without collisions, and the simulated payment rail behaves like the real one down to test cards for specific decline codes. Stripe treats the sandbox as a first-class product feature, and it shows in how much of the ecosystem tests against Stripe by default.

Salesforce has been pulled in the same direction by its own agent ambitions. Scratch orgs and sandboxes now support Agentforce, and its Testing Center exists precisely because Salesforce concluded you cannot ship an autonomous agent that was never rehearsed against production-shaped metadata, validation rules, and automations. Notice the causality: the vendor shipping agents forced the vendor to invest in agent-grade test environments.

Then the ladder drops off. QuickBooks has a sandbox, but Intuit's own guidance concedes that webhooks, error codes, and rate-limiting behavior can differ from production — the exact state-dependent, side-effect-laden behaviors an agent most needs to rehearse. And Gmail, one of the most common agent surfaces through its MCP server, has no sandbox tier at all. There is no "scratch inbox" product. Your agent's email tooling gets tested against a real mailbox belonging to someone, or it doesn't get tested.

The result is an incoherent safety posture: the same agent, with the same architecture and the same eval discipline, is well-tested on its Stripe tools, partially tested on its QuickBooks tools, and untested on its Gmail tools. No amount of internal engineering rigor fixes this, because the missing piece is on the vendor's side of the boundary.

What a Real Rehearsal Tier Requires

Suppose you take this seriously and build the best environment tier currently possible. It has three layers, and each substitutes for the ones a vendor failed to provide.

Layer one: vendor sandboxes where they exist, wired in as first-class environments. Your agent's tool configuration should carry an environment axis the same way your services do — the same send_invoice tool resolving to sandbox credentials in staging and live credentials in prod. This sounds obvious and is widely not done; a remarkable number of agent stacks have exactly one credential set per tool, which means every test anyone runs is a production test. OWASP's MCP guidance says the quiet part: high-risk operations need runtime controls precisely because most deployments have no earlier tier where those operations could have been exercised.

Layer two: a record-replay proxy at the tool boundary for everything else. Not inside the agent framework — at the boundary, where it sees every tool call regardless of which model or orchestrator produced it. Recording production traffic (with PII scrubbed) gives you regression fixtures for free. Replaying with injected faults — 429s, partial failures, out-of-order webhooks — gives you the chaos testing that no vendor sandbox offers.

The discipline that makes layer two work: treat cassettes like dependencies with freshness dates, and re-record on a schedule. A stale fixture is a mock wearing a trench coat.

Layer three: synthetic tenants for the surfaces with no sandbox. For Gmail-shaped problems, this means a real account you own, populated with generated-but-realistic data — threads with attachments, labels, half-finished drafts, the accumulated sediment of actual use. Synthetic tenants are expensive to build and maintain, which is why almost nobody has them, and they are also the only honest answer for write-action testing where the vendor provides nothing. The population problem is real: an empty test org tells you little, because most interesting agent failures come from state — the duplicate contact, the closed period, the thread with a legal-hold label.

None of this is exotic technology. It's the same environment engineering we've done for internal services for two decades, applied at a boundary we've historically treated as someone else's problem. The reason it feels novel is that pre-agent integrations were low-volume and human-triggered — a person clicked a button, one API call happened, and if it failed a person saw the error. Agents changed the volume, the autonomy, and the blast radius simultaneously, and the testing infrastructure didn't move.

Scratch Orgs Are Coming to Your Category, Ready or Not

Here's the forward-looking claim: sandbox quality is about to become a procurement criterion for SaaS, driven by agents.

The pressure mechanism is straightforward. Every SaaS vendor is currently racing to be agent-accessible — shipping MCP servers, publishing tool schemas, courting the integrations. But an agent-accessible API without an agent-testable environment is a liability the buyer inherits: enterprises deploying agents will start asking vendors the questions they already ask about SSO and audit logs. Does your platform have a sandbox with production-shaped data? Do webhooks fire in it? Do rate limits match? Can I clone a tenant's configuration — not their data — into a scratch environment?

Salesforce already made this pivot because Agentforce forced its hand. Stripe made it a decade ago because payments punish sloppiness. The long tail of B2B SaaS has not, and the gap will become visible the first time a customer's agent does something expensive in an environment that had no cheaper tier to rehearse in. "Scratch org" is going to stop being Salesforce jargon and start being a checklist item — as unremarkable a demand as "SAML support" is today.

Until that pressure does its work, the practical guidance for agent teams is unglamorous: inventory your tools by environment tier. For each write-capable tool, answer one question — where can this action be rehearsed? Vendor sandbox, replay proxy, synthetic tenant, or nowhere. The tools in the "nowhere" column are your real risk register, far more than anything in your prompt-injection threat model. Wrap them in the runtime controls — approval gates, allowlists, spend caps — that exist to compensate for a missing environment, and be honest that a compensating control is what they are.

The world has no staging environment yet. The teams that internalize this stop asking "is our agent smart enough to ship?" and start asking the older, better question: "where did this thing rehearse?"