126 posts tagged with "security"

The worst place to discover that you shipped an AI agent is in an incident review. Not the design doc, not the architecture review, not the change ticket — the incident review, where a lawyer is asking why a customer's account got a refund nobody authorized, and an engineer is scrolling through a service that was last meaningfully reviewed eleven months ago, and there, forty lines into a function called enrichTicket, is a model call that reads the customer record, decides on a resolution, and calls the billing API. Nobody diagrammed it. Nobody approved it as an agent, because to the person who wrote it, it wasn't one. It was "just a helper."

This is shadow AI, and it has grown teeth. The first wave was employees pasting company data into consumer chatbots — a data-leakage problem, bad but bounded. The second wave is agents: model calls wired into internal tools, reading real data and taking real actions, sitting inside services that were approved for something else entirely. Roughly half of employees admit to using AI tools their employer never sanctioned, and a striking share of that usage comes from the top — directors and executives are among the biggest culprits, not the interns. When the same instinct reaches your codebase, you don't get a leaked spreadsheet. You get an autonomous actor with production credentials that nobody signed off on.

The uncomfortable part is that shadow agents aren't created by reckless people. They're created by good engineers doing exactly what you asked them to do: ship value fast, reuse existing infrastructure, don't file a ticket for every small change. The governance gap isn't a discipline problem. It's a definitional one — your review process doesn't have a category for what they built.

Your onboarding checklist screens for a lot of things: background checks, conflict-of-interest disclosures, signed IP assignment agreements, a laptop with the right MDM profile. It does not screen for the thing your new senior engineer actually walked in with — a personal AI subscription whose memory contains eighteen months of accumulated context from their previous employer. Architecture debates, incident retrospectives, unreleased product names, the exact shape of the query optimizer they just spent two years building. Not as documents they exfiltrated, but as ambient context an assistant absorbed one conversation at a time, synthesized in the background, and will happily draw on when they ask it a question at their new job.

The same boundary fails in the other direction on the way out. When that engineer leaves you, your offboarding runbook will revoke their SSO, kill their OAuth tokens, and wipe their laptop. It will not — because it cannot — touch the personal ChatGPT or Claude account where your architecture decisions, your incident history, and your roadmap now live as memory entries in a tenant you don't own, can't audit, and can't wipe. Employment has always leaked knowledge through human memory, and the law made peace with that. What's new is a second, machine-grade memory that travels with the person: searchable, persistent, verbatim in places, and invisible to both employers on either side of the transition.

Every security team has a mental model of typosquatting: an attacker registers requets and waits for someone to fat-finger requests. It works, but it's a scattershot bet on human clumsiness. Slopsquatting is worse, because the "typo" isn't random. Language models invent plausible-but-nonexistent package names in predictable, repeatable patterns — and attackers can query the same models you use, harvest the names they invent, and register exactly those packages on PyPI and npm. The hallucination becomes a preorder. Your coding agent, running with autonomous install rights, is the customer who picks it up.

This isn't hypothetical. The largest study of the phenomenon generated 2.23 million code samples across sixteen models and found that 19.7 percent of recommended packages didn't exist — 205,474 unique fabricated names. And when a security researcher registered one of the most commonly hallucinated Python packages as a harmless empty shell, it was downloaded more than 30,000 times in three months and ended up in the install instructions of a major tech company's open-source repository. The supply chain attack that vibe coding made possible is already running its proof of concept.

You built an agent that checks supplier prices every morning. It runs a real browser, clicks real buttons, and does exactly what a diligent employee would do — just faster and at 6 a.m. Then one day it stops working. No error in your code. The supplier's site now returns a challenge page, then a 403, then nothing at all. Your agent didn't break. It got classified.

Here is the uncomfortable symmetry every agent builder eventually confronts: the web's defenses were tuned by a decade of scraper wars, credential-stuffing attacks, and inventory-hoarding bots — and your legitimate agent is statistically indistinguishable from all of them. Automated traffic passed 50% of all web traffic in 2024, and roughly 37% of the total is classified as malicious. The immune system that evolved to fight that flood does not care about your intentions. It cares about your fingerprint, and your fingerprint says "bot."

Somewhere in your company's data classification policy, there is a table. Raw customer text — support tickets, medical notes, chat transcripts — sits in the "sensitive" row, wrapped in encryption requirements, access controls, and deletion SLAs. And then there is your vector store, holding embeddings of that exact text, classified as... nothing. Derived data. Anonymous math. Just floats.

That classification is wrong, and it is wrong in a way that is now experimentally demonstrated. Inversion attacks can reconstruct the original text from its embedding — in the best-studied setting, recovering 32-token inputs exactly in 92% of cases, including full names from clinical notes. If an attacker with your vectors can read your customers' words, your vectors inherit the sensitivity of those words. The regulators have started saying this out loud, and most retrieval architectures are not ready for what follows: a deletion request that has to reach every index, every snapshot, and every derived artifact — not just the row store.

There is a number that should end the debate about whether your company has a shadow AI problem. In a single month, security researchers watching enterprise browser traffic logged over 155,000 copy actions and more than 313,000 paste actions into generative AI tools. Not requests. Not page views. Copy-and-paste events — the precise gesture an engineer makes when they lift a stack trace out of a production log and drop it into a chatbot to ask why the service is crashing.

The uncomfortable truth is that this already happened. Your employees did not wait for the AI policy committee to convene. They pasted the customer list, the contract draft, the proprietary database schema, and the half-broken function into a consumer chatbot months ago, and they did it from a personal account your IT department cannot see. The question is no longer how to prevent shadow AI. It is how to govern an activity that is already pervasive, mostly invisible, and not going to stop.

A security incident review opens with a question the team cannot answer: how did the agent learn the name of the tool it just called? The audit trail shows a tools/call for a tool whose name does not appear in any tools/list response the harness logged. The MCP server cheerfully accepted the call and executed it. The model, asked in a postmortem to explain where the tool name came from, offers no answer because there is none — it guessed, and the guess landed on a real action.

This is the failure mode at the seam between two assumptions that look compatible on paper. The client treats the tool list as a contract that names the surface area of authority it has been granted. The server treats the tool list as a snapshot of what is currently available, free to grow when the world grows. Between those two views, the LLM is a bridge that does not know the difference.

A new engineer joined the team on Monday. By Wednesday, she had a working local agent setup: an MCP server bridged to the company's deployment API, pointed at staging, talking to her editor. The onboarding doc walked her through the OAuth flow. The token she pasted into the server's environment file was the one her teammate had emailed her — the same token the CI pipeline uses to ship to staging. By Friday, she had joined the team for a working session at a coworking space.

The MCP server was still running. Bound to 127.0.0.1. No authentication. The token was loaded into the process. She didn't think about it because she was not using it. But any tab that visited any website that day could speak to her local server through her own browser. So could any other laptop on the coworking wifi, because she had not noticed that the server was actually bound to 0.0.0.0. The OAuth token your CI pipeline uses to push to staging was now reachable by anyone who could trick a browser into making a request to a local IP — which, in 2026, is a one-pop-up problem.

This post is about that class of failure: the gap between "I'm developing on my laptop" and "my laptop is a server reachable by adversaries." MCP servers, by design, sit right in that gap. Most teams have not noticed.

The design document said each tool gets its own OAuth token, scoped to the minimum permissions that tool needs. The implementation stored tokens keyed by (user_id, provider). Both statements were true on the day v1 shipped, because there was exactly one tool per provider. The day a second tool against the same provider went live, the design document was still true and the storage layer silently invalidated it.

Six months later, a security review traced an incident back to that line of schema. A calendar-reader tool, compromised through a prompt injection in an event description, had successfully called events.delete on the user's primary calendar. The reader had never been granted that scope. The writer had. The token store didn't distinguish between them.

This is the failure mode where a per-provider key shape silently aggregates privilege across tools that share a provider — and the architectural realization that OAuth scope is a property of a token, not a property of a tool.

The security review said the agent acts "on behalf of" the user. The OAuth token said something else, and the audit log agreed with the token.

A small distinction in language did a lot of architectural work nobody noticed. "On behalf of" is the language a security review reaches for when it wants to capture an arrangement where the agent is a delegate, recognizable as a delegate, and constrained by being a delegate. "Act as" is the runtime behavior when the agent holds a token indistinguishable from the user's own and is therefore the user as far as every downstream system can tell. These two phrases describe completely different threat models. A typical enterprise OAuth integration ships the second one and prices it as the first.

A PII redactor that runs only on inbound traffic is a one-way valve installed at the wrong end of the pipeline. It catches user-submitted names, emails, and account numbers before they reach your logs. It does nothing about the model's own outputs — the place where the same model is now actively assembling text that may contain those same identifiers, drawn from RAG retrievals, tool returns, conversation history, or content the user pasted from another tenant's data. Every team I've watched ship an input-side redactor has a follow-up ticket in the backlog labeled "output-side parity." Most of those tickets never close, because no incident surfaces the gap for six months, and after six months the ticket has accumulated enough re-prioritization to look like a feature request rather than a missing half of a security control.

The failure mode is invariant: input redaction is treated as the canonical control because it is the easier engineering problem and the easier audit story. You wrote a regex set, you ran a labeled benchmark, you proved precision and recall on a fixed corpus, you shipped it behind a feature flag, and the security review accepted it as the PII boundary. The output side has none of that benefit. The model's response is generative, the surface area is unbounded, and the test methodology — "what should it not say in any of infinitely many contexts" — is structurally harder than "what should we strip from a known input." So the team that ships the inlet treats the outlet as future work and the future never arrives until a customer reports another customer's email landing in their transcript.

A Spanish customer complained that her annual renewal had been billed a day early. The support ticket bounced through three queues before it landed in front of an engineer who recognized the smell: a date-formatting regression, European cohort only. He ran git log against the date-formatting module and found nothing. The module had not been touched in eleven days. What had been touched, eleven days earlier, was its package.json — a lodash bump from 4.17.20 to 4.17.22, opened by a security agent, approved by the on-call, merged without comment.

Two lines above the version string, in the same file, was a comment written eighteen months ago: // do not upgrade — breaks the snapshot tests in date-formatting, see FRONT-2418. The security agent had not read it. Or, more precisely: the security agent had read the entire file, but its prompt instructed it to find vulnerable version strings, not to weigh the comments around them. The comment was load-bearing institutional knowledge. The agent treated it as scenery.

This is a coordination failure between two systems that did not know they were colliding. The security agent was doing its job. The original engineer who wrote the comment had done his job. The feature-development agent that respected the pin every time it touched the file was doing its job. Nobody had decided whose job it was to mediate between them.