GraphRAG vs. Vector RAG: The Architecture Decision Teams Make Too Late

Most teams discover they need GraphRAG six months too late — after they've already explained to users why the AI got the relationship wrong, why it confused two entities that share similar embeddings, or why it confidently cited a document that contradicts the actual answer. Vector RAG is genuinely good at what it does. The problem is that teams treat it as good at everything, and keep piling on retrieval hacks when the underlying architecture has hit a mathematical ceiling.

Fewer than 15% of enterprises have deployed graph-based retrieval in production as of 2025. This is not because the technology is immature. It's because the failure signals for vector-only RAG are subtle: the system runs, the LLM responds, and only careful inspection reveals that the retrieved context was plausible but wrong.

What Vector RAG Actually Does — and Where the Ceiling Is

Vector RAG converts documents into high-dimensional embeddings and retrieves content by nearest-neighbor similarity at query time. For semantic search — "find documents related to this topic" — it works remarkably well. The model captures conceptual similarity in ways that keyword search cannot.

But similarity is not the same as correctness, and the embedding architecture has hard mathematical limits. A 512-dimensional embedding model reliably degrades past 500,000 documents. Even a 4,096-dimensional model hits scaling problems at 250 million documents. More data doesn't help — the information bottleneck is the vector itself. You are compressing the semantics of a document into a fixed-size number, and eventually that compression loses precision that matters.

The deeper problem is structural: vector embeddings represent documents in isolation. When chunks are embedded and indexed, the relationships between entities across documents disappear. An agent might retrieve chunk A ("Company X invested heavily in AI infrastructure") and chunk B ("AI infrastructure carries significant regulatory exposure"), but never surface the connection that X's specific investment created that specific exposure. The retriever returns two separately relevant documents; the LLM must infer the link. Sometimes it does. Often it doesn't, or worse, invents a plausible link that isn't supported by the corpus.

Four query patterns expose this ceiling faster than any benchmark:

Exact identifier lookups. When a user queries "Error 221," vector search may return "Error 222" documentation — semantically close, factually wrong. Embeddings compress the numerical distinction into noise.

Multi-constraint queries. "Blue trail-running shoes, size 10, under $100" generates a single vector that averages the constraints. The retriever may return red shoes at $150 if they match on "trail-running." Graph traversal can satisfy multiple attribute constraints simultaneously without averaging them away.

Rare terminology and domain jargon. Product SKUs, legal codes, financial instrument identifiers — embedding space compresses these into the gravitational field of nearby concepts. A knowledge graph preserves them as distinct, addressable nodes.

Multi-hop relational queries. "How does new regulation X affect our supply chain vendor Y's compliance posture?" requires traversing from regulation → affected industries → specific vendors → vendor compliance history. No amount of chunking strategy surfaces this through semantic similarity.

How GraphRAG Works

GraphRAG takes a different approach. Instead of embedding document chunks, it extracts entities and relationships from text and organizes them as a graph — nodes for entities, edges for relationships. At query time, it translates user intent into graph traversal operations rather than similarity lookups.

Microsoft introduced GraphRAG in 2024 with a hierarchical architecture: extract entities and relationships, cluster them into communities using the Leiden algorithm, and generate summaries of each community. Queries can operate in two modes: global search (synthesis across community summaries, good for broad questions about the entire corpus) or local search (fan out from specific entity nodes, good for fact lookups about named things).

The empirical results are striking. On enterprise benchmarks, GraphRAG achieves 86% accuracy on multi-hop tasks where vector RAG scores 32% — a 54-percentage-point gap. On schema-bound queries requiring complex aggregations, vector RAG accuracy drops to 0% while graph-based retrieval hits 90%. When queries involve 10 or more entities, vector RAG accuracy degrades to zero; GraphRAG sustains above 70%.

These numbers are real. They're also cherry-picked for the worst case. On simple semantic search — "find documents discussing topic X" — vector RAG and GraphRAG perform comparably, and the graph adds overhead without benefit. The architecture decision isn't "which is better." It's "which is better for your query distribution."

The Query Patterns That Tell You You've Outgrown Vector Search

Before you invest in graph infrastructure, you need to know whether your actual users are running the queries that benefit from it. Measure this directly.

Build an evaluation set of 50–100 queries that represent real user intent from your production logs. Include the expected answers. Run recall@k and precision@k against your current retrieval system. When recall@k drops below 70% on a curated set that matters, retrieval is your bottleneck — not the LLM, and not your prompts.

Specific signals to look for:

• Queries involving 3+ named entities that require reasoning about how they relate
• Time-indexed relationship queries ("what changed in the contract between X and Y after 2024?")
• Contradiction detection ("does our documentation contradict itself on this point?")
• Influence chain queries ("what downstream systems are affected by this API change?")
• Comparative relationship queries ("how does Company A's approach compare to Company B's on this regulation?")

If these patterns appear frequently in your evaluation set and your recall is suffering, the vector architecture is not the right fit. If your queries are primarily "summarize the relevant sections about topic X," you may not need the graph at all.

The Migration Path: Don't Rip and Replace

The 2025 consensus among practitioners is clear: don't rebuild from scratch. Successful migrations layer graph capabilities on top of existing vector infrastructure rather than replacing it.

Phase 1: Instrument first. Before changing architecture, add proper observability. Track recall@k and precision@k by query type. Log which retrieval results the LLM actually used versus ignored. Identify where the failure is happening — 73% of RAG system failures occur at the retrieval stage, not during generation. You cannot fix what you cannot measure.

Phase 2: Hybrid baseline. The highest-leverage step before introducing graphs is combining what you already have. Add BM25 keyword search alongside vector retrieval. Add a cross-encoder reranker that rescores the combined candidates. This three-layer approach (vector + BM25 + rerank) yields 15–25% accuracy improvement on most enterprise corpora with no graph infrastructure required. Many teams stop here and find it sufficient.

Phase 3: Selective graph augmentation. If phase 2 is insufficient for your multi-hop queries, add graph-based context enrichment selectively. The pattern: use vector search to identify top-K candidate documents, then use graph traversal to enrich those candidates with related entities and relationships. This "vector-first, graph-enrich" hybrid performs better than either approach alone on complex queries, while keeping graph infrastructure out of the hot path for simple semantic lookups.

Tools like Neo4j support both vector indexes and property graph traversal in a unified backend. LlamaIndex's PropertyGraph abstraction and LangChain's graph integrations make the implementation surface manageable. You don't need a separate graph database standing next to your vector store if you choose the right backend.

Phase 4: Full GraphRAG where justified. Reserve the full hierarchical community-detection approach for corpora where global synthesis across the entire document set is required. The original Microsoft approach costs $20–500 to index a typical enterprise corpus. Microsoft's LazyGraphRAG variant, released in June 2025, reduces this to under $5 by deferring community summarization to query time — at the cost of 2–8 additional seconds per query.

Building the Graph: The Hidden Complexity

Knowledge graph construction is where most teams underestimate the effort. The extraction pipeline has three stages, each with its own failure modes.

Entity extraction identifies named entities in text: people, organizations, products, regulations, dates. Modern approaches use pretrained NER models fine-tuned for your domain. The challenge is domain-specific entities — product SKUs, internal project codenames, custom identifiers — that generic models miss. LLM-based extraction catches more but costs 10–50x more than running a specialized NER model.

Relation extraction identifies how entities relate: "Company X acquired Company Y," "Regulation A governs activity B," "System C depends on System D." The quality of this step determines whether your graph enables multi-hop reasoning or produces a disconnected node collection. Dependency-parsing approaches using libraries like spaCy are cheap and fast; LLM-based triple extraction is more accurate on complex sentences but expensive at scale.

Graph assembly merges extractions across the corpus, handles entity de-duplication ("Apple" = "Apple Inc." = "AAPL"), and builds the connectivity that makes traversal useful. This is often more work than the extraction itself. Without principled de-duplication, you get a fragmented graph where the same real-world entity appears as dozens of disconnected nodes.

The practical recommendation for teams starting out: use dependency-based extraction with spaCy for the bulk of the work, fall back to LLM extraction only for ambiguous multi-entity sentences, and budget significant time for de-duplication rules. The graph is only as useful as its connectivity.

When GraphRAG Hurts

The accuracy improvement numbers are real. The failure modes are also real.

Over-indexed graphs produce noise. If your extraction pipeline runs on everything and filters nothing, the graph fills up with low-quality relations that create false traversal paths. An entity appearing in a passing reference gets connected to the same cluster as an entity that is structurally central to the corpus. Graph traversal amplifies this noise because bad edges multiply.

Naive Cypher queries are slow. Graph traversal on large graphs without careful query design can take seconds. Vector search with HNSW returns results in milliseconds. If you're routing all queries through graph traversal, you will hit latency problems that vector-only search never had.

GraphRAG hurts on simple semantic search. For queries that vector RAG handles well — "find documents about topic X" — adding graph traversal increases latency and can actually hurt precision by adding tangentially related context. The query routing logic that decides when to use graph enrichment versus vector-only retrieval is not trivial to get right.

Community re-computation is not free. As your corpus grows and changes, the hierarchical clustering that enables global search must be partially or fully recomputed. Incremental updates to nodes and edges are O(1); invalidating and regenerating community summaries is not. This operational cost is often invisible during initial deployment.

The Decision Framework

GraphRAG is the right architectural investment when:

• Your evaluation set shows recall@k below 60% on representative multi-hop queries
• Your domain fundamentally involves relationship chains: supply chain analysis, legal clause dependencies, organizational hierarchies, medical comorbidities, financial instrument relationships
• Your corpus is large (1M+ documents) and changes incrementally rather than in bulk
• Users frequently ask synthesis questions that span many documents rather than retrieval questions about specific topics

Vector-only RAG is sufficient when:

• Queries are primarily semantic ("find relevant sections on topic X")
• Your corpus is updated in bulk and re-embedding is feasible
• Acceptable accuracy is above 70% on single-hop factual lookups
• Latency requirements are strict (sub-100ms retrieval)

The hybrid approach — vector + BM25 + reranker, with selective graph enrichment for known multi-hop query patterns — is the right starting point for most production systems. Ship the baseline, instrument it properly, identify the failure class, and layer graph infrastructure only where the measurement shows retrieval is failing for structural reasons rather than semantic ones.

What the Cost Curve Now Looks Like

The objection that killed many GraphRAG projects in 2024 was indexing cost. Microsoft's original approach cost $33,000 or more to index large enterprise corpora. That number was accurate at the time and immediately made GraphRAG impractical for most teams.

The cost curve has changed. LazyGraphRAG achieves comparable quality at under $5 per corpus by deferring community summarization to query time. HippoRAG, from OSU's NLP group, delivers multi-hop reasoning accuracy at 10–30x lower cost than the hierarchical community approach. PathRAG reduces context window usage by 44% through principled graph path pruning, which cuts both latency and token costs. OG-RAG adds ontology grounding that reduces hallucinations by 40% without the full indexing pipeline.

Teams that wrote off GraphRAG in 2024 should revisit the economics. The architecture is not the same system it was twelve months ago.

The Diagnostic You Should Run This Week

If you have a production RAG system and haven't measured retrieval performance separately from generation performance, that's the highest-leverage thing you can do. Pull 100 queries from your logs that users engaged with or complained about. Annotate the expected answers. Run recall@k at k=5 and k=10. Break the queries into categories by type — single-entity lookup, multi-entity synthesis, relationship chain, comparative analysis.

Where recall is above 80%, your retrieval is healthy. Where it drops below 60%, you've found a failure class. That failure class tells you whether adding BM25 would help (keyword-heavy queries), whether a reranker would help (semantic search where top results aren't the most useful), or whether graph enrichment is needed (structural relationship queries).

Most teams, when they do this exercise, discover that graph architecture is necessary for a specific slice of their query distribution — not all of it. That's the right answer. The architectural decision isn't binary. It's about knowing which queries need graph semantics and routing them accordingly.

The teams making this decision now, before they've spent a year explaining retrieval failures to users, are the ones who will avoid the most expensive version of this lesson.