3 posts tagged with "llm-security"

A security team I talked to recently found that their prompt and response fields were being shipped, in full, to a third-party SaaS logging backend they had never signed a Data Processing Agreement with. The fields contained customer medical summaries, Stripe secret keys accidentally pasted by support agents, and the full text of a confidential acquisition memo that someone had asked an internal assistant to summarize. Nothing was encrypted in the payload. Nothing was redacted. The retention was 400 days. The integration was set up during a hackathon by a well-meaning engineer who pip install-ed the vendor's SDK, dropped in an API key, and shipped.

This is the AI observability leak. Every LLM app team ends up wanting tracing — you cannot debug prompt regressions or non-deterministic agent loops without it — so one of LangSmith, Langfuse, Helicone, Phoenix, Braintrust, or a vendor AI add-on ends up in the stack. The default setup captures the entire request and response. That default is, for most production workloads, a compliance violation waiting to be discovered.

A startup CTO checked their OpenAI dashboard one morning and found a $67,000 invoice. Their normal monthly bill was $400. Nothing in their product had changed — no viral launch, no new feature, no marketing push. What had changed is that an attacker fingerprinted their endpoint, harvested a leaked key from a build artifact, and resold the inference at 40-60% below retail to buyers who paid in crypto. The startup paid the bill while the attacker pocketed the spread.

This is not the typical free tier abuse story SaaS founders tell each other. The typical story goes: a few power users abuse generous trials, churn rates spike, you tighten the limits, and unit economics recover within a quarter. That playbook is dead for AI products. The math broke when your unit cost per anonymous request stopped being effectively zero, and the abuse playbook scaled the moment your generosity could be liquidated for cash.

Prompt injection is the number one vulnerability in the OWASP Top 10 for LLM applications — and the gap between how engineers think it works and how attackers actually exploit it keeps getting wider. A 2024 study tested 36 production LLM-integrated applications and found 31 susceptible. A 2025 red-team found that 100% of published prompt defenses could be bypassed by human attackers given enough attempts.

The hard truth: the naive defenses most teams reach for first — system prompt warnings, keyword filters, output sanitization alone — fail against any attacker who tries more than one approach. What works is architectural: separating privilege, isolating untrusted data, and constraining what an LLM can actually do based on what it has seen.

This post is a field guide for engineers building production systems. No CTF-style toy examples — just the attack patterns causing real incidents and the defense patterns that measurably reduce risk.