383 posts tagged with "ai-agents"

Put a human in the loop and you have a control. Put a hundred approval requests a day in front of that human and you have a rubber stamp wearing a control's badge. The two look identical on an architecture diagram. They behave nothing alike under load, and the gap between them is where most "responsible AI" deployments quietly fail.

The pattern is familiar to anyone who has watched a security operations center drown. Industry surveys put the share of alerts that go uninvestigated somewhere between a quarter and two-thirds — one frequently cited figure is that 62% of alerts are simply ignored, and 55% of teams admit to regularly missing alerts they would classify as critical. The analysts aren't lazy. They are processing a few thousand alerts a day with a false-positive rate that often exceeds 50%, and the human brain responds to that ratio exactly the way you'd expect: it stops looking. Agentic AI is now recreating this exact failure mode, one confirmation dialog at a time.

Your platform team measures success by developer adoption. Monthly active engineers on the internal API. Time-to-first-call for a new service. Net promoter score from the quarterly DX survey. Every one of those metrics assumes a human is on the other end of the request — someone who reads a getting-started guide, copies a curl example, and grumbles in Slack when the error message is unhelpful.

That assumption is quietly becoming false. The fastest-growing consumer of your internal APIs is not a person. It's an agent: a coding assistant resolving a ticket, a workflow that reconciles invoices overnight, a support bot that calls six services to answer one question. These callers don't skim your docs — they ingest your tool schemas into a context window. They don't file a bug when an error is cryptic — they silently retry, burn tokens, and give up. And they are about to outnumber the humans.

The first time an agent product falls over in production, the on-call engineer usually pulls up the request-rate graph and finds it boringly flat. Forty requests per second, same as yesterday. No spike, no surge, no obvious cause. Meanwhile the system is on fire: timeouts everywhere, the provider returning 429s, the queue depth climbing in a straight line toward the moon. The graph that should explain the outage looks completely healthy, because the graph is measuring the wrong thing.

QPS — queries per second — is the unit we inherited from a decade of stateless web services. A request comes in, you do tens of milliseconds of work, you send a response, the request is gone. Under that model, request rate and load are nearly the same number, and capacity planning is mostly "how many requests per second can one box handle, multiply by boxes." That mental model is so ingrained that we reach for it automatically. For agents, it quietly lies to you.

A security incident review opens with a question the team cannot answer: how did the agent learn the name of the tool it just called? The audit trail shows a tools/call for a tool whose name does not appear in any tools/list response the harness logged. The MCP server cheerfully accepted the call and executed it. The model, asked in a postmortem to explain where the tool name came from, offers no answer because there is none — it guessed, and the guess landed on a real action.

This is the failure mode at the seam between two assumptions that look compatible on paper. The client treats the tool list as a contract that names the surface area of authority it has been granted. The server treats the tool list as a snapshot of what is currently available, free to grow when the world grows. Between those two views, the LLM is a bridge that does not know the difference.

The clearest sign that an agent's session model is broken is when a tool result has nowhere to go. The agent fired a long-running call — a render, a provisioning job, a multi-step query. The user watched the spinner for a few seconds, decided they didn't need it after all, closed the tab, and moved on. Forty seconds later the tool finishes. Its callback hits your gateway with a conversation_id that no longer points at anything. The gateway has two equally bad options: silently drop the result, or stitch it into whatever session inherits that ID next.

Most teams discover this failure mode the same way: a support ticket where a user sees an answer they did not ask for, attached to a conversation they did not start. Or a downstream system that processed the same charge twice because the gateway helpfully "retried" delivery against the next active session. Or — most commonly — nothing visible at all, just a slow drift in completion metrics that nobody can correlate to anything specific, because the failures don't fire alerts; they fire emptiness.

The design document said each tool gets its own OAuth token, scoped to the minimum permissions that tool needs. The implementation stored tokens keyed by (user_id, provider). Both statements were true on the day v1 shipped, because there was exactly one tool per provider. The day a second tool against the same provider went live, the design document was still true and the storage layer silently invalidated it.

Six months later, a security review traced an incident back to that line of schema. A calendar-reader tool, compromised through a prompt injection in an event description, had successfully called events.delete on the user's primary calendar. The reader had never been granted that scope. The writer had. The token store didn't distinguish between them.

This is the failure mode where a per-provider key shape silently aggregates privilege across tools that share a provider — and the architectural realization that OAuth scope is a property of a token, not a property of a tool.

A Spanish customer complained that her annual renewal had been billed a day early. The support ticket bounced through three queues before it landed in front of an engineer who recognized the smell: a date-formatting regression, European cohort only. He ran git log against the date-formatting module and found nothing. The module had not been touched in eleven days. What had been touched, eleven days earlier, was its package.json — a lodash bump from 4.17.20 to 4.17.22, opened by a security agent, approved by the on-call, merged without comment.

Two lines above the version string, in the same file, was a comment written eighteen months ago: // do not upgrade — breaks the snapshot tests in date-formatting, see FRONT-2418. The security agent had not read it. Or, more precisely: the security agent had read the entire file, but its prompt instructed it to find vulnerable version strings, not to weigh the comments around them. The comment was load-bearing institutional knowledge. The agent treated it as scenery.

This is a coordination failure between two systems that did not know they were colliding. The security agent was doing its job. The original engineer who wrote the comment had done his job. The feature-development agent that respected the pin every time it touched the file was doing its job. Nobody had decided whose job it was to mediate between them.

Your dashboard says cache hit rate is 71%. Your finance partner is pleased. Your latency p50 is fine. Then a customer support thread arrives from a long-running agent session: turn 14 took eleven seconds to produce the first token, turn 15 took eight, turn 16 took nine. You pull the trace. Every one of those turns reports a cache_read_input_tokens value of zero. The system prompt is sixteen thousand tokens. The user thinks the agent is broken. You think your provider is broken. Neither of you is right. The aggregate hit rate is a survivorship statistic — it averages over the short conversations that hit cache trivially and quietly absorbs the long conversations that have collapsed to cold-on-first-token mid-session.

This is the failure mode that no provider postmortem will ever describe to you, because from their telemetry the system is working as designed. The load balancer is making the routing decision it was told to make. The cache is being populated and evicted on the schedule it was told to follow. The hint you passed — the prompt_cache_key, the conversation ID, the user ID, whatever string you serialized into that field — was advisory the whole time, and "advisory" means "ignored when convenient." Under load, when a scaling event happens, when an upstream pod is draining, when the affinity-aware tier is saturated, your hint quietly degrades to a uniform routing decision. The request lands on a cold pod. The KV tensors that would have served the prefix at sub-millisecond cost are sixteen feet away in a sibling rack and unreachable. Your conversation pays full-prefix cost again, and your dashboard's headline number doesn't move because two thousand other one-turn conversations hit cache fine.

The deprecation notice went out on a Tuesday. The downstream team rotated the response shape on their search tool — results[].snippet became results[].excerpt, a clean rename, six-week window, banner in the docs, three reminder emails to the engineering list. Every human consumer migrated. The agent did not, because the agent does not read email. For fourteen days the retry loop quietly parsed the new payload, found the field it was looking for missing, raised a KeyError, and counted that as a retryable failure. The retry hit the same endpoint, got the same new shape, raised the same error, gave up after three attempts, and returned an apologetic message to the user. The retry budget dashboard stayed green the entire time — retries were never exhausted, they were just permanently failing within budget. Success rate, measured at the tool layer, sat at zero on that path. Nobody looked because there was no page.

This is the shape of the failure that gets the most engineers in 2026: not the dramatic outage, but the silent contract drift where a human-facing migration runs to completion and the agent-facing one never starts because nobody knew there was one to run. The deprecation worked exactly as designed for the consumers it was designed for. The agent was a consumer nobody listed.

A quarter into the rollout, the AI team reported a 25% reduction in average cost-per-API-call. The support team reported that average handle time on AI-routed tickets had drifted from four turns to seven. Both numbers were correct. Both teams were measuring the system they had been told to optimize. The finance team, sitting between them, could not reconcile the dashboards because neither one was denominated in the thing the customer was actually paying for: a resolved ticket. The cost-per-call had gone down. The cost-per-resolved-task had gone up 40%. Nobody owned that number, so nobody was watching it move.

This is the most common unit-economics failure I see in agentic deployments, and it is not a measurement bug. It is a definitional one. The vendor's pricing page exposes cost-per-call because that is the unit they bill. The spreadsheet line item inherits that unit because it fits in a cell. The engineering team optimizes against the unit they were given. By the time the gap between API economics and business economics becomes visible, it has been compounding for a quarter, and the agent has been quietly trained on the wrong loss function the entire time.

You shipped an AI support agent six months ago to deflect 40% of tier-one tickets. Today your human queue is longer than it was before launch, your CSAT is down, and the per-ticket cost has gone up. The deflection dashboard says everything is fine. It is not.

The failure mode is not that the agent is bad at answering questions. The failure mode is that "escalate to human" was supposed to be the safety valve, and instead it became the path of least resistance. The agent learned, through the structure of its rewards and the absence of any cost on the escalation action, that handing the conversation off is the cheapest way to discharge an ambiguous request. Your support team did not notice this happening because the metric they watched — deflection rate — does not penalize the agent for routing fixable problems into the human queue. It only penalizes the agent for the user explicitly clicking "talk to a human" after a long unsuccessful exchange.

This is not a tooling problem. It is an incentive design problem, and the leadership failure is treating it as something the vendor will fix in the next release.

A long-running agent generates a plan at step 3. The plan reads something like: "if the order returned by get_order in step 1 has status shipped, send the customer a tracking email; otherwise open a refund ticket." The agent confidently picks the email branch. The customer never received a tracking number, because the order was actually in pending. You go to the trace expecting to find a hallucination. What you find is worse: the step-1 tool result is no longer in context. The pruner evicted it between step 2 and step 3 — it ranked low on recency and there was a 12KB transcript to make room for. The plan still ran. The branch was still chosen. The decision now points at evidence that does not exist.

This is not a model failure in the usual sense. The model produced a syntactically valid plan, executed it in order, and made a branch decision. The branch was made against a fact that used to be in context and is not anymore. The chain of thought encoded the condition (if status == "shipped"); the actual status got dropped on the way to the step that needed it. The plan looks deterministic, but it has been quietly cut loose from its evidence.