28 posts tagged with "platform-engineering"

At 4:50 PM on a Friday, someone on the data team kicks off an eval sweep: forty thousand prompts against the company's shared model deployment, scheduled to finish over the weekend. At 5:10 PM, the customer-facing chat assistant starts timing out. The on-call engineer spends two hours staring at dashboards that show the provider returning 429s before someone thinks to ask who else is using the account. Nothing is broken. The system is doing exactly what it was configured to do, which is nothing, because nobody configured it to do anything.

This is the shape of a new class of incident, and it has a property that makes it nastier than an ordinary outage: there is no bug to fix. The eval sweep was legitimate work. The chat assistant's traffic was legitimate work. The failure is that two teams with different urgency profiles were drawing from one undifferentiated pool of inference capacity, and the pool had no opinion about who mattered more. Capacity allocation stopped being an infrastructure detail the moment your company had more than one team shipping against the same provider account — it became a political problem, and the pager inherited it.

Two years ago, standing up an AI center of excellence was the responsible move. Nobody knew how to evaluate a model, procurement had no idea what an inference contract should look like, and legal wanted one throat to choke. Concentrating the ten people who understood any of it into a central team was obviously correct.

Today that same team is the reason your product engineers wait six weeks to change a prompt. The CoE reviews every system-message edit, owns the only eval harness in the company, and gates model upgrades behind a committee that meets biweekly. Teams have noticed. They ship on personal API keys, run evals in notebooks the CoE never sees, and paste customer data into whatever tool answers fastest. You did not prevent shadow AI — you created it, and gave it a governance body to hide from.

Every internal LLM platform goes through the same arc. In month one, inference is free: the platform team eats the bill, product teams experiment wildly, and everyone celebrates adoption curves. By month six, the bill has grown 10x, finance is asking pointed questions, and the platform team discovers that three teams account for 80% of spend — one of them running a nightly batch job nobody remembers approving. The instinctive response is to install a meter and start charging. That instinct, applied naively, is how you kill your platform.

Here's the uncomfortable truth: per-token chargeback punishes exactly the behavior you built the platform to encourage. The team prototyping an agent that might transform your support workflow burns tokens like a furnace — agentic workloads consume 5 to 30 times more tokens per task than a simple chat completion. Bill them list price from day one and they stop prototyping. Meanwhile, the team running a mature, optimized feature pays pennies and looks virtuous. You've built a pricing system that taxes learning and rewards stagnation.

Your platform team measures success by developer adoption. Monthly active engineers on the internal API. Time-to-first-call for a new service. Net promoter score from the quarterly DX survey. Every one of those metrics assumes a human is on the other end of the request — someone who reads a getting-started guide, copies a curl example, and grumbles in Slack when the error message is unhelpful.

That assumption is quietly becoming false. The fastest-growing consumer of your internal APIs is not a person. It's an agent: a coding assistant resolving a ticket, a workflow that reconciles invoices overnight, a support bot that calls six services to answer one question. These callers don't skim your docs — they ingest your tool schemas into a context window. They don't file a bug when an error is cryptic — they silently retry, burn tokens, and give up. And they are about to outnumber the humans.

A compliance program is a description of the systems your company had on the day the auditor signed off. The systems your company has today are a different set, and the gap is the surface area of every release that shipped a new persistent store between then and now. The deletion guarantee you sold your customers is a guarantee against the first set, and the regulator who eventually asks about it will be asking about the second.

The failure mode is not a bug in the deletion code. The deletion code is correct. The saga fans out across every storage system named in the data inventory, calls each one's erasure endpoint, collects a receipt per system, and reports success when every receipt comes back signed. The saga is doing exactly what it was built to do. The problem is that the saga is iterating over a list of storage systems that was true eighteen months ago, and the agent platform team shipped a long-term memory feature six months ago that nobody added to the list.

A platform team I know spent two quarters building a model registry. It had everything the org chart asked for: a promotion workflow from dev to staging to prod, a CODEOWNERS-style approval matrix, lineage tracking, eval-score gates, a deprecation policy with a 30-day window, and a Backstage tile that showed which version of every model was live in which service. They cut a launch announcement, ran a brown bag, and added a row to the compliance binder.

Six months later, the highest-traffic agent in the company was running on a model card whose "owner" field still pointed at someone who had left, whose eval score was from a benchmark the team had since deprecated, and whose "approved by" name was the platform tech lead — who had never used that agent, never read its eval set, and had pressed approve at 11:43pm on a Thursday because the producer had pinged him in DMs saying the launch was tomorrow.

The registry was not broken. The promotion gates fired. The audit log was intact. Everything the launch announcement had promised was true. And the org had less real oversight of its production models than it had had eighteen months earlier, when the same decisions were made by an ML engineer reading the eval output by hand before pasting the model URI into a config file.

A platform team ships a quality improvement. An inference-time layer reads the user's recent interactions and silently nudges the response style: more formal here, more terse there, more technical when the history suggests an engineer is asking. The A/B test shows an aggregate satisfaction lift of a couple of points. The launch post goes out under the heading "smarter responses, no API changes required." Nobody flips a flag in the API. Nobody updates the docs. Nothing in the response payload indicates which persona the model just adopted.

Six weeks later an enterprise customer files a support ticket that says, "your model is worse than you advertised." Their internal eval suite — running the same prompts your team published benchmarks against — scores eight points lower. Your team's first move is to verify prompt parity. Prompts match exactly. Decoding parameters match. The model version string matches. The divergence traces to the personalization layer, which infers a "thin-history default persona" for the customer's freshly-provisioned test account and a richer one for the long-lived user accounts your benchmarks were measured against. The conversation about whether the personalization is a feature or a bug stops being a product decision and becomes a contract negotiation.

Open a production agent and count what is in the context window before the user has typed a single character. There is a system prompt the platform team owns. There are tool definitions — forty of them, maybe more — each carrying a name, a description, a JSON schema, field-level docs, and a handful of enums. There is a block of retrieved examples that the search team added because few-shot helped one eval. There are six lines of safety instructions from trust and safety, four lines of formatting rules from the design team, and a paragraph of domain glossary that someone added during an incident and nobody removed.

Add it up and the agent boots with 30,000 tokens of overhead. On a connected setup with three MCP servers, that number is routinely far worse — one widely cited measurement put three servers at 143,000 of a 200,000-token budget, 72% of the window consumed before the conversation starts. None of it is wrong. Every line was added by someone solving a real problem. And that is exactly why the context window is being destroyed.

The pitch sounded responsible. "Let's not hardcode OpenAI everywhere — we'll put a thin abstraction in front, then we can swap providers if we need to." Two years later, that thin abstraction is a service with its own deploy pipeline, its own SRE on-call, an eval gate that blocks bad prompts, a semantic cache that saves seven figures a year, a retry policy with provider-specific backoffs, an observability schema every dashboard depends on, and a key vault holding the credentials for six model vendors. Every AI feature in the company terminates there.

It is also, almost by accident, the single point of failure with the worst blast radius in the stack. When the primary LLM provider goes down — and in 2025 OpenAI was tracked having 294 outage events since January, with Anthropic logging 184.5 hours of total customer impact in December alone — the gateway routes around it and most users never notice. When the gateway itself dies, every AI feature in every product simultaneously stops, the failover that was supposed to fire never gets a chance, and the postmortem opens with "the abstraction layer we built to insulate us from provider outages was the outage."

The most strategic AI investment in your company is probably a Slack bot one engineer built on a Friday afternoon. It answers "how do I get a staging credential" or "which on-call is responsible for the auth service" or "what's the runbook for a stuck deploy," and it has saved more engineering hours than the entire customer-facing AI roadmap that absorbs three quarters of your model spend, your safety review queue, and your launch comm bandwidth.

The org chart doesn't reflect this. The OKR doc doesn't reflect this. Nobody is the PM. Nobody is the EM. The bot survives because the engineer who built it still answers the GitHub issues, and the value compounds quietly while every customer-facing feature ships behind a six-week safety review and a launch readiness checklist that exists because the customer might churn.

A team I worked with shipped a "smart" date extractor last quarter. The model parsed natural-language phrases like "next Tuesday" and "two weeks from the 14th," ran in production behind a feature flag, and cost about three cents per request at the chosen tier. Six weeks later, a backend engineer wandered into a design review and mentioned, casually, that the company already had a date parser. It had been written in 2019, lived in a utility module nobody on the AI team had read, handled 99.4% of the same inputs at sub-millisecond latency, and ran for free. The AI feature did not get pulled. It got rationalized — "the model handles the long tail" — and the team moved on, having shipped a more expensive, slower, less accurate version of something the company already owned.

This is not a one-off story. It is the dominant failure mode for AI features inside companies older than the AI team. The pattern repeats: a smart classifier duplicates a regex pipeline written years ago, a retrieval system fetches a vendor list that an internal service has been maintaining as a typed table, an agent learns to extract entities a parser already extracts deterministically. The AI feature ships with a quality bar lower than the deterministic system it didn't know existed, and the team who built the deterministic system finds out at a cross-team meeting.

Six months after launching their first AI agent, most engineering organizations discover they have fifteen of them. Not because anyone planned a fleet — because each team solved a real problem and shipped. The customer support team built a triage agent. The data team built a report-generation agent. Platform engineering built a runbook agent. Infrastructure built three more. None of them share auth, logging, tooling, or evaluation methodology. Tokens are bleeding from a dozen provider accounts and nobody can tell you which agent is responsible.

This is the moment that separates engineering organizations that can scale AI from those that can't. The answer is not to slow down agent development — it's to run a portfolio audit before entropy makes consolidation impossible.