65 posts tagged with "retrieval"

A support engineer pastes a customer ticket into your internal assistant. The question is sharp: "Does our API support multi-region writes on the free tier?" The assistant comes back instantly, citing a chunk it retrieved with 0.91 cosine similarity. The answer is yes. The chunk is from a landing page written by marketing in 2023 to win a head-to-head against a competitor. Engineering removed multi-region writes from the free tier eighteen months ago and posted a terse internal RFC that nobody linked from a customer-facing page. The RFC is also in the vector store. It scored 0.74.

The assistant didn't hallucinate. It retrieved the highest-scoring document and faithfully grounded its answer in the text. The retriever did its job. The job was wrong.

Your operations agent answers a question about the upcoming launch by quoting a Slack message that says "we'll ship tomorrow." The agent treats that as a present-tense plan and starts writing comms. The message was posted six weeks ago. The ship happened. The retrieval pipeline pulled the right chunk by every metric you measure — semantic similarity to "launch date," top-1 confidence above your threshold, source channel matching the project — and the agent built a plan on a sentence that meant something only inside the meeting where it was written.

The bug is not in the model. The bug is that tomorrow is not a date. It is a pointer to a clock, and the clock the message was written against is not the clock the agent is reading it on. Your retrieval pipeline indexed the body of the message and discarded the frame.

You embedded the knowledge base eighteen months ago. The model has not changed. The chunks have not changed. The index is healthy, the latency is fine, the recall dashboard is a flat line at 0.86. And yet support is quietly pasting the wrong article links into ticket replies, the sales bot keeps surfacing a deprecated SKU when a prospect asks about the new one, and an internal user just told you the assistant "feels dumber" without being able to say why.

Nothing broke. Your embeddings aged. The word post used to mean blog post in your domain; now half the corpus uses it for a Slack post, a forum post, and a job posting, and your eighteen-month-old vectors still treat it as one concept. The model that encoded those vectors never saw the new senses, never saw the new product names, never saw the rebrand, never saw the regulation that introduced three new terms your customers now use without thinking. The retrieval system answers the question it knows how to answer, which is no longer the question your users are asking.

A customer asks your support agent how to do something. The agent retrieves three documentation chunks with high relevance scores, synthesizes a confident answer, and walks the customer through a five-step procedure that ends on a button that hasn't existed for four months. The customer files a ticket. The on-call engineer pulls the eval suite, finds it green, pulls the retrieval traces, finds them green too — the model didn't hallucinate, it faithfully quoted documentation describing a feature your product team renamed in the last quarterly release.

This is the failure mode I want to name: not a hallucination, not a retrieval miss, but a phantom inventory problem. Your retrieval corpus is a snapshot of a product surface that no longer exists. The vector store doesn't know the product changed. The eval suite doesn't know either. The only system that consistently catches it is the support ticket queue, and by the time a ticket is filed the customer has already been told to click a button that isn't there.

A user asks your assistant what the company's parental leave policy is. The bot returns 12 weeks, with a citation. The cited document was the right answer in 2023; HR posted an update last quarter that took it to 16. Both versions are in your knowledge base. Cosine similarity scored the 2023 version 0.87 and the 2024 version 0.84, because the older page has the cleaner phrasing and fewer hedges. The fresher document loses by three percentage points and the user gets a wrong answer that looks audited.

This is the freshness-relevance tradeoff, and the uncomfortable part is that it has no clean solution at query time. If you weight recency, you bias retrieval toward whatever was edited yesterday — which in most knowledge bases is the noisy, high-churn surface area that should not be the source of truth. If you don't weight recency, you ship answers grounded in documents that were superseded months ago. There is no single global knob that gets both right, and most teams discover this only after a few embarrassing answers leak past their eval suite.

A user asks your support bot when the refund window closes. The bot answers "60 days" with cheerful confidence and a citation. The policy page that says "60 days" was deleted from the CMS three months ago. The new policy is 14. Nobody on your team knows the bot is wrong until a customer escalates.

This is a retrieval cascade failure: the document is gone from the source of truth, but its embedding is still in the index, still ranking high on cosine similarity, still feeding the model a ghost. RAG pipelines treat embedding indexes as caches of source content, but most teams build the cache without building the invalidation. Inserts get all the engineering attention. Deletes get a TODO comment.

Your RAG pipeline hit 80% retrieval accuracy on the eval set. The team ships it. Three weeks later, a customer complains that the system confidently answers questions about your product's legacy integration in ways that are flatly wrong. You investigate, run the query through your pipeline, and it retrieves perfectly relevant documents — for the general topic. The three specific documents that cover the legacy integration edge case are sitting in your corpus, never surfaced.

That 80% number was real. It was also nearly useless as a signal for what just happened.

Your RAG system has a bug that doesn't throw exceptions. It doesn't spike your error rate. It doesn't show up in your latency dashboards. Instead, it quietly delivers confident, plausible-sounding answers that are wrong — and nobody notices for weeks.

This is the data contract problem in RAG: your ingestion pipeline is the source of truth for everything downstream, but it has no schema enforcement, no freshness guarantees, and no alerting when the shape of the world changes underneath it. Every time an upstream data source adds a field, a chunking parameter shifts, or an embedding model gets updated, your retrieval quality silently degrades.

Eighty percent of enterprise RAG projects experience critical failures in production. The most insidious of those failures don't announce themselves.

Your RAG system passed all the retrieval benchmarks. Precision looks solid. The LLM-as-judge eval scores are green. And yet, somewhere in your index, there is a document describing an API endpoint that was deprecated eight months ago, a pricing tier that no longer exists, and a compliance policy that was superseded by new regulations in Q3. Your retriever has no idea. Semantic similarity has no concept of time.

This is the knowledge half-life problem: the silent failure mode where RAG systems appear healthy on every metric you're measuring while serving increasingly stale decisions to users. Seventy-three percent of organizations report accuracy degradation in RAG deployments within 90 days — not from poor retrieval architecture or embedding model quality, but from knowledge staleness that no one modeled as a reliability concern.

Your RAG pipeline looks solid on paper: chunking is clean, the vector store is indexed, latency is acceptable. But users keep complaining that the results are wrong — not completely wrong, just slightly wrong in ways that matter. The retrieved passage discusses the right concept but from the wrong time period. It covers the right topic but from the wrong jurisdiction. It mentions the right product but is missing the inventory signal that would make it actually useful.

This is the embedding fine-tuning gap. Generic embedding models are trained to encode semantic similarity — the property of two texts meaning roughly the same thing. That's not the same as relevance. Relevance is domain-specific, context-sensitive, and often invisible to a model trained on web-scale generic corpora.

A team at a product company built a brainstorming assistant for their marketing department. They added RAG over their document corpus — campaign briefs, brand guidelines, competitor analyses — figuring the richer context would produce better ideas. Usage dropped within three weeks. The qualitative feedback: outputs felt "too safe," "too predictable," "like it just remixed our existing stuff." They removed retrieval from the brainstorming feature. Ideas improved. Engagement recovered.

This pattern repeats more often than practitioners admit. Retrieval-augmented generation has become the default architecture for grounding LLM outputs in facts, and for factual tasks it earns that default. But for generative tasks — ideation, creative writing, novel solution generation — adding a retrieval layer can silently cap the ceiling of what your model produces. Not because retrieval is broken, but because it's working exactly as designed.

Most teams building AI agents discover the same problem on their fifth sprint: the agent can't reliably pick the right tool anymore. At ten tools, it mostly works. At twenty, accuracy starts to slip. At fifty, you're watching the agent call search_documents when it should call update_record, and the logs offer no explanation. The usual reaction is to tweak the tool descriptions — add more context, be more explicit, rewrite the examples. This occasionally helps. But it misses the root cause: flat embedding retrieval is architecturally wrong for large tool inventories, and better descriptions cannot fix an architectural problem.

Tool selection is retrieval, and retrieval has known scaling limits. Understanding those limits — and the structured metadata patterns that work around them — is what separates agent systems that hold up in production from ones that require constant babysitting.