GraphRAG vs. Vector RAG: When Retrieval Needs to Be a Graph Problem

If you've shipped a RAG system and it handles single-fact lookups fine but falls apart on questions like "which vendors did we drop after the Q3 security review, and why," you've hit the ceiling of vector-only retrieval. That's not a chunking problem or an embedding-model problem. It's a structural mismatch: the question requires traversing relationships between entities, and a vector index has no concept of relationships — only similarity.

This post covers when that mismatch shows up, how GraphRAG addresses it, and how to build a minimal graph-augmented retrieval layer without committing to a full graph database migration.

The problem: similarity search has no notion of connection

A vector index answers one question well: "what text is semantically close to this query?" It does this by embedding chunks and the query into the same space and ranking by cosine similarity (or dot product). That's the right tool when the answer lives inside a single chunk or a small set of independently relevant chunks.

It breaks down on three common query shapes:

1. Multi-hop questions. "Who approved the vendor that caused the outage in March?" requires connecting an outage record → a vendor → an approval record. No single chunk contains that chain, and top-k similarity search retrieves chunks that mention "outage," "vendor," or "approval" independently — not the path between them.
2. Aggregation over entities. "How many customers churned after we changed the pricing page in June?" requires enumerating entities and a relationship (churn events tied to a time window), not finding a passage that states the answer.
3. Global/summarization questions. "What are the main themes across all support tickets this quarter?" requires reasoning over the whole corpus's structure, not the top 10 most similar chunks to a query embedding.

Increasing k, reranking, or using a better embedding model helps at the margins but doesn't fix the underlying issue: vector search retrieves by similarity, and these queries need retrieval by relationship.

What GraphRAG actually does

GraphRAG (a term Microsoft Research popularized with their 2024 paper and open-source implementation, since adopted more broadly) does two things differently from vector RAG:

1. It builds an explicit entity-relationship graph from your corpus at ingestion time, usually by running an LLM extraction pass over chunks to pull out (entity, relationship, entity) triples — e.g., (Acme Corp, was_approved_by, Jane Chen). This graph is stored separately from (or alongside) your vector index.

2. It retrieves via graph traversal in addition to, or instead of, similarity search. Given a query, you first identify the relevant entities (via NER or similarity search against entity names), then walk the graph outward from those entities to pull in connected facts, then feed that structured context to the LLM instead of — or in addition to — raw chunk text.

Microsoft's implementation also does community detection (clustering the graph into topic communities) and pre-summarizes each community, which is what enables the "global" summarization use case — you're not walking the graph at query time for those, you're hitting a precomputed summary of a graph region.

Do you actually need it?

Building and maintaining a knowledge graph is real infrastructure cost: extraction pipelines, entity resolution, graph storage, and a retrieval layer that knows how to traverse it. Most teams don't need the full Microsoft GraphRAG pipeline. Use this as a decision guide:

Signal	Vector RAG is enough	Consider GraphRAG
Query shape	"What does X mean / say about Y"	"How is X connected to Y through Z"
Document structure	Mostly independent documents (FAQs, docs, policies)	Entity-dense records (org charts, contracts, incident logs, CRM data)
Answer locality	Answer lives in one or two chunks	Answer requires chaining facts across documents
Query volume vs. corpus size	Large corpus, narrow queries	Smaller corpus, but queries require global/aggregate reasoning
Team capacity	Want to ship fast, iterate on chunking/reranking	Can own an extraction + graph maintenance pipeline

A useful heuristic: if you can't answer the failing query by handing a human five of your top-10 retrieved chunks, and the fix is "the human would need to read three specific other documents and connect them," you have a graph problem.

Building a minimal GraphRAG layer

You don't need Neo4j or a managed graph database to get the core benefit. A lightweight version — entity extraction into a plain graph structure (e.g., networkx in memory, or a Postgres table with (source, relation, target) rows) — covers most mid-size use cases.

Step 1: Extract entities and relationships at ingestion

import json
from openai import OpenAI

client = OpenAI()

EXTRACTION_PROMPT = """Extract entities and relationships from the text below.
Return JSON: {"entities": [{"name": str, "type": str}],
"relationships": [{"source": str, "relation": str, "target": str}]}
Only extract relationships explicitly stated or clearly implied in the text.

Text:
{chunk}
"""

def extract_graph(chunk: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": EXTRACTION_PROMPT.format(chunk=chunk)}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Run this over every chunk during ingestion, alongside your normal embedding step. Keep the source chunk ID attached to each extracted triple so you can trace back to the original text.

Step 2: Store the graph

import networkx as nx

G = nx.MultiDiGraph()

def add_to_graph(graph_data: dict, source_chunk_id: str):
    for rel in graph_data["relationships"]:
        G.add_edge(
            rel["source"], rel["target"],
            relation=rel["relation"],
            source_chunk=source_chunk_id,
        )

For anything beyond a few hundred thousand nodes, move this to Postgres with a (source, relation, target, source_chunk_id) table and a couple of indexes — you don't need a specialized graph database to traverse 1-2 hops.

Step 3: Entity resolution (don't skip this)

Raw extraction gives you "Acme Corp," "Acme," and "Acme Corporation" as three separate nodes. Without resolution, your graph fragments and traversal misses edges. A cheap fix: embed entity names, cluster near-duplicates above a similarity threshold, and merge nodes — or run a second LLM pass that dedupes entity lists per document batch. This step is where most homegrown GraphRAG implementations quietly fail; budget real time for it.

Step 4: Query-time retrieval — hybrid, not graph-only

def retrieve(query: str, k_vector: int = 8, hops: int = 2) -> list[str]:
    vector_hits = vector_search(query, k=k_vector)  # your existing pipeline

    seed_entities = extract_query_entities(query)  # cheap NER pass or LLM call
    graph_context = []
    for entity in seed_entities:
        if entity in G:
            for _, target, data in nx.bfs_edges(G, entity, depth_limit=hops):
                graph_context.append(
                    f"{entity} {data.get('relation', '?')} {target}"
                )

    return vector_hits + graph_context

Feed both the vector hits (raw passages) and the graph context (structured facts) into the prompt. Don't replace vector search — combine them. Most production GraphRAG systems are hybrid: vector search for "what's relevant," graph traversal for "how is it connected."

Comparison: vector RAG vs. graph-augmented RAG

Dimension	Vector RAG	GraphRAG
Ingestion cost	Embed chunks	Embed chunks + LLM extraction pass + entity resolution
Query latency	One vector search	Vector search + graph traversal (2-3x latency typically)
Multi-hop reasoning	Poor — relies on lucky chunk co-occurrence	Native — explicit traversal
Global/summarization queries	Poor without extra map-reduce logic	Good if community summarization is built
Maintenance	Re-embed on content change	Re-extract + re-resolve entities on content change
Failure mode	Misses connected facts silently	Extraction errors propagate as wrong edges

Common pitfalls

• Extracting relationships from every chunk in isolation loses cross-chunk context. A pronoun reference ("he approved it") extracted from a chunk without its antecedent produces garbage edges. Extract with a sliding window or full-document context where possible, not one 500-token chunk at a time.
• Skipping entity resolution. This is the single biggest source of "GraphRAG didn't work" reports. If "Q3 2025" and "third quarter of 2025" are different nodes, your traversal silently fails.
• Treating graph traversal as a replacement for vector search instead of a complement. Pure graph retrieval loses the semantic fuzziness that makes vector search good at "what's roughly related to this." Hybrid retrieval consistently outperforms either alone in practice.
• No confidence scoring on extracted edges. LLM extraction hallucinates relationships, especially on ambiguous text. If you can, have the extraction step cite the source sentence for each triple, and surface that provenance at query time so you can debug wrong answers back to a specific bad edge.
• Building the full Microsoft GraphRAG pipeline (community detection, hierarchical summarization) when you only need 2-hop traversal. Community summarization is expensive to build and maintain — reserve it for genuinely corpus-wide summarization use cases, not as a default.

Next steps

Start by auditing your actual failing queries — pull the last 20 "bad answer" logs from your RAG system and classify them as single-fact-miss (fix with better chunking/reranking) versus multi-hop-miss (graph problem). Most teams find that only 10-20% of failures are genuinely graph-shaped, which is often enough to justify a minimal hybrid layer like the one above, but not enough to justify a full graph database migration on day one.

If you're building or debugging retrieval systems and want hands-on practice with extraction pipelines, entity resolution, and hybrid retrieval design, this is exactly the kind of system-design work we cover in Joinloop's AI engineering cohort — with real corpora, not toy datasets.