178 posts tagged with "rag"

Your RAG eval dashboard is green. Faithfulness is 0.91, answer relevance is 0.88, and the LLM-as-judge harness you spent two sprints building says the system is doing fine. Meanwhile, a user just asked a question whose answer sits in a document your retriever never surfaced, and your model wrote a confident, well-structured, entirely useless response about something adjacent. The judge scored it highly. It read well. It was grounded in the passages it did get. It just answered the wrong question with material that had nothing to do with what the user needed.

This is the quiet structural flaw in how most teams evaluate retrieval-augmented generation: they grade the essay and never check whether the student was handed the right book. A RAG system is two machines bolted together — a retriever that decides what the model gets to see, and a generator that decides what to do with it. Almost every eval harness in production measures only the second machine. The first one, the one that actually determines the ceiling on answer quality, runs unmonitored.

There is a specific kind of outage that never shows up as an outage. The service stays green, latency is flat, error rates are zero, and search quietly starts returning garbage. This is what happens the moment you point a new embedding model at an index built by the old one. Nothing crashes. The results just stop making sense.

The reason is geometry. An embedding model doesn't assign fixed coordinates to a concept — it defines a space, and the same sentence lands in a completely different location depending on which model drew the map. A vector produced by last year's model and a query embedded by this year's model are not "close" or "far." They are measured against different rulers. Cosine similarity between them is a number, and the number is meaningless.

So when someone files a ticket titled "upgrade to the new embedding model," they have not filed a config change. They have filed a full data migration that happens to be disguised as a one-line diff. Treat it like a library bump and you ship the silent outage.

Most teams threat-model the wrong plane. They harden the chat box — rate limits, input validation, a jailbreak classifier watching what the user types — and they treat everything the model reads as inert. The wiki page, the support ticket, the scraped webpage, the calendar invite, the PDF someone uploaded: data, not instructions. Background material for the model to summarize, not commands for it to obey.

That assumption is the vulnerability. The moment your agent retrieves content and drops it into the context window, that content is executing with the same authority as your system prompt. There is no privilege boundary between "here are your instructions" and "here is a document to consider." It's all just tokens, and the model was trained to follow instructions wherever they appear.

The web's economics rest on an assumption so old nobody wrote it down: the thing loading the page has eyeballs. A human arrives, an ad impression fires, an affiliate cookie drops, an analytics event attributes the visit — and that chain of tiny monetization events pays for the content. Every part of that chain is now breaking at once, because a growing share of your site's readers aren't people. They're agents, and an agent doesn't see ads. It extracts the answer, hands it to a user somewhere else, and leaves nothing behind but a log line.

This isn't a distant publisher problem you can watch from the engineering sidelines. If you build anything on retrieval — a RAG pipeline, an agent that browses, a product that summarizes the web — you are on the demand side of a market whose supply side just discovered it's been giving away inventory for free. The correction is underway, it has infrastructure and standards behind it, and it lands on your architecture as a new cost line and a new failure mode: upstream sources that were free and open last quarter going paywalled, licensed, or dark this quarter.

The Hadoop generation learned one lesson so thoroughly it became a reflex: moving data is expensive, so move the computation to the data. Every MapReduce scheduler, every HDFS block placement decision, every "data locality" dashboard existed to serve that principle. Then, somewhere between the rise of managed model APIs and the agent boom, we quietly inverted it — and nobody repriced the decision.

Look at what a modern agent loop actually does. It retrieves a stack of documents from a vector store, pulls a repo snapshot from object storage, collects tool results from half a dozen internal services, concatenates all of it into a context window, and ships the whole payload to a model endpoint that usually lives in a different VPC, often a different region, sometimes a different cloud. Then it does it again on the next turn. And the next. Your context has mass, and you are paying freight on every hop.

Your agent answered a customer's billing question with last quarter's pricing, and the postmortem will blame the model. It shouldn't. The prompt was assembled correctly, the retrieval scored well, the model reasoned soundly over everything it was given — and everything it was given was true three days ago. Somewhere between the CRM export, the docs sync, and the vector index rebuild, "current state of the world" quietly became "state of the world as of Tuesday," and nothing in your stack was measuring the difference.

Data engineers solved this class of problem years ago. A downstream dashboard consuming ten upstream tables gets lineage, freshness checks, and an on-call rotation that pages when the nightly job slips. The context window your agent consumes is the same thing — a materialized view joined from docs, tickets, code, CRM, and memory — except nobody owns the join, nothing measures its staleness, and when it serves yesterday's truth the failure gets filed as "the model hallucinated."

Somewhere in your company's data classification policy, there is a table. Raw customer text — support tickets, medical notes, chat transcripts — sits in the "sensitive" row, wrapped in encryption requirements, access controls, and deletion SLAs. And then there is your vector store, holding embeddings of that exact text, classified as... nothing. Derived data. Anonymous math. Just floats.

That classification is wrong, and it is wrong in a way that is now experimentally demonstrated. Inversion attacks can reconstruct the original text from its embedding — in the best-studied setting, recovering 32-token inputs exactly in 92% of cases, including full names from clinical notes. If an attacker with your vectors can read your customers' words, your vectors inherit the sensitivity of those words. The regulators have started saying this out loud, and most retrieval architectures are not ready for what follows: a deletion request that has to reach every index, every snapshot, and every derived artifact — not just the row store.

A support engineer pulled up your RAG console to debug a complaint. The customer had asked "what does my account look like right now," the answer had come back coherent and confident, and it had been about somebody else's account entirely. The top-3 retrieved chunks all belonged to other customers. The engineer ran the same query against a fresh corpus snapshot to rule out indexing lag. Same result. Then she ran it against a snapshot from six months ago, before the privacy redactor had shipped. The right customer's chunk came back at rank 1.

The redactor was working as designed. Every name was a [NAME], every email an [EMAIL], every account number an [ACCOUNT]. The legal team had a clean audit trail and the security team had a closed compliance ticket. What nobody on either team had modeled was that those sentinels, dropped into the same syntactic slots across millions of documents, were being seen by the embedding model as ordinary tokens — tokens that co-occurred more reliably with each other than any real content did. The redactor had not just removed information. It had added a new, very strong signal that every redacted document shared and nothing else did.

The chunker started prepending [line N] to every chunk. The eval went green. Every citation the model produced after that day pointed to the paragraph one position before the actual evidence, on every document, in the regulated industry the product serves. The team did not find out from the eval. The team found out from an auditor who looked at the cited sentence, read it, and pointed out that it contradicted the claim it was supposed to support.

This is the kind of regression that survives a code review, a manual QA pass on three sample documents, and a feature-flag rollout. None of those checks were wrong in isolation. They were all asking the same question — does a citation appear where one is expected — and none of them were asking the question the auditor asked, which is whether the citation points at the sentence the claim came from. The gap between those two questions is where the off-by-one lived for as long as it lived.

What makes this failure mode worth a separate write-up is not the bug itself. Off-by-one errors are old news. The interesting part is that the failure was produced by two systems that continued to agree on the structure of an integer while silently disagreeing about what the integer meant.

A RAG agent answers a customer's regulatory question with a tidy paragraph and a citation. The verification layer fetches the URL, sees a 200 OK, ticks the box, and ships. Six months later a compliance audit pulls the transcript, clicks the same link, and finds a page that now says the opposite of what the agent quoted. The URL is fine. The quote is fine in the transcript. The two no longer match. The customer's compliance officer asks whether the agent fabricated the quote, and the team cannot prove it didn't, because the only surviving evidence of what the URL used to say is the agent's own assertion of what it said.

This is not a hallucination in the usual sense. The model retrieved real content, faithfully extracted a real sentence, and emitted a real URL that still resolves. Every link-checker on earth would call this citation valid. The audit fails anyway, because the verification layer was measuring the wrong property. Reachability is not fidelity. A URL is a pointer to a mutable document under someone else's editorial control, and the moment the document changes, every transcript that quoted it becomes a hallucination report waiting to happen.

Your finance team built a beautiful AI cost dashboard. Token spend, sliced by feature. Embedding spend, sliced by provider. Every quarter, the per-feature pane gets reviewed in a leadership meeting and somebody asks why the support-chat workflow is up 12%, and a product manager has a defensible answer. Every quarter, the per-provider pane gets reviewed in an infra meeting and somebody asks why OpenAI is up 8%, and a platform engineer has a defensible answer. And every quarter, the line that actually doubles your AI bill — the corpus re-index — lands in a third bucket called "infrastructure" that nobody reviews because nobody owns it.

That bucket is where forty percent of your AI spend goes to die unattributed. The teams who could have optimized it never see it. The teams who see it can't tell you which feature it serves. The dashboard is honest about every cost it can explain and silent about the cost it can't, which is exactly the cost that matters most.

The most expensive embedding bug a RAG system can ship is the one where nothing in your repository changes. Your retrieval code is the same. Your index is the same. Your query path is the same. And one Tuesday in week six, somebody notices that the answers used to be better.

The provider posted a sunset notice for the embedding family your index was built against twelve months ago. The platform team filed it in a deprecations dashboard with a year of runway and moved on. The sunset path wasn't a hard cutoff — it was a quiet quality regression where the deprecated endpoint started routing to a "compatibility" successor that returned vectors in the same dimensionality and a subtly different semantic geometry. Query embeddings began drifting against the corpus you embedded a year ago. Recall@10 on your standing eval slid by 47% over six weeks. The team only traced it back when an unrelated quality dashboard crossed a threshold, dragging a senior engineer into a root-cause exercise that ended at an embedding endpoint no one on the call had touched in a year.