3 posts tagged with "cloud-costs"

At 4:50 PM on a Friday, someone on the data team kicks off an eval sweep: forty thousand prompts against the company's shared model deployment, scheduled to finish over the weekend. At 5:10 PM, the customer-facing chat assistant starts timing out. The on-call engineer spends two hours staring at dashboards that show the provider returning 429s before someone thinks to ask who else is using the account. Nothing is broken. The system is doing exactly what it was configured to do, which is nothing, because nobody configured it to do anything.

This is the shape of a new class of incident, and it has a property that makes it nastier than an ordinary outage: there is no bug to fix. The eval sweep was legitimate work. The chat assistant's traffic was legitimate work. The failure is that two teams with different urgency profiles were drawing from one undifferentiated pool of inference capacity, and the pool had no opinion about who mattered more. Capacity allocation stopped being an infrastructure detail the moment your company had more than one team shipping against the same provider account — it became a political problem, and the pager inherited it.

Fifteen years ago, we learned to fear the N+1 query: an ORM that looked innocent in code review would fire one query for a list and then one more query per row, and a page that should have made two database calls made two hundred. We fixed it with eager loading, batching, and a generation of linters. Then we built AI agents, and we shipped the same bug at a much more expensive layer.

A single agent task — "reconcile these invoices," "triage this incident" — routinely makes dozens of serial tool calls. Each one is a full network round trip: agent to tool, tool to data store, data back through the tool, result serialized into the model's context, another inference pass to decide the next call. If your inference runs in one cloud and your data lives in another, every one of those hops crosses a metered, high-latency boundary. The dominant cost term of your agent system is no longer the model. It's the geography.

Nobody itemized this. Latency budgets were written per call, egress was a rounding error on the storage bill, and the model invoice got all the scrutiny. Meanwhile the tool loop quietly became the chattiest client your infrastructure has ever served.

The Hadoop generation learned one lesson so thoroughly it became a reflex: moving data is expensive, so move the computation to the data. Every MapReduce scheduler, every HDFS block placement decision, every "data locality" dashboard existed to serve that principle. Then, somewhere between the rise of managed model APIs and the agent boom, we quietly inverted it — and nobody repriced the decision.

Look at what a modern agent loop actually does. It retrieves a stack of documents from a vector store, pulls a repo snapshot from object storage, collects tool results from half a dozen internal services, concatenates all of it into a context window, and ships the whole payload to a model endpoint that usually lives in a different VPC, often a different region, sometimes a different cloud. Then it does it again on the next turn. And the next. Your context has mass, and you are paying freight on every hop.