Graph RAG in Practice: When RAG Falls Short on Interconnected Domain Knowledge
An AI assistant that answers questions from a company's own documents has become one of the most obvious applications in business today. The technique behind it is called Retrieval-Augmented Generation, or RAG for short, and for a large share of cases it is the right choice.
There is, however, a starting situation in which this standard approach reliably falls short. This occurs precisely where knowledge consists of rules, exceptions, and areas of responsibility, that is, in insurance, healthcare, finance, and anywhere an answer has consequences.
This article uses a concrete case from Swiss healthcare to show where this limit lies, how a knowledge graph shifts it, and what the approach costs in effort. The first part is aimed at everyone who decides on such projects. The second describes the technical implementation in detail.
The starting point was a statement made by a company in the first meeting: an internally tested assistant delivered unusable results. A RAG chatbot had been built with Copilot Studio and another with Google's AI Studio, and the results were unsatisfactory. With Copilot Studio the answer was simply wrong, and with Google's AI Studio the answer took over 3 minutes. A challenge worth examining more closely.
How RAG Works
A chatbot with a standard LLM initially answers only from its training knowledge. It knows nothing about a company's internal documents, and whatever it doesn't know, it fills in with something that sounds plausible if in doubt - this is how hallucinations arise. RAG solves this by giving the model the relevant passages from its own knowledge source for each question.
To do this, documents are first broken down into smaller sections (chunks). Each section is converted into what is known as an embedding, a numerical representation of its meaning. Sections with similar meaning lie close together in this meaning space, even if they use different words. These representations are stored in a vector database.
When someone asks a question, it too is converted into an embedding. The system searches for the sections that lie closest to the question and passes them, together with the question, to the language model, which formulates the answer from them.
The added value is considerable. The answer is based on verified, current, and company-specific content instead of general training knowledge, and the sources can be cited. This works reliably as long as one condition is met: the question must be answerable using embeddings that are semantically close to one another in vector space.
The Starting Situation
The company operates several software systems for different user groups. Depending on the language, the documentation spans between 300 and 650 pages, is available in three languages, and a significant portion of the information is contained in screenshots rather than in running text.
Three properties of this knowledge base are decisive.
- It is distributed. Content resides in a content management system, in SharePoint, and in Confluence, without consistent linking between related rules.
- It is role-dependent. The same question has a different correct answer depending on the function of the person asking, because permissions and responsibilities vary.
- It is relational. A single statement is rarely complete on its own. Rules have exceptions, exceptions have conditions, and these are often found in a different document than the rule itself.
Against this backdrop, the service desk answers recurring questions by hand every day. An assistant that covers the standard cases is the obvious relief. This exact approach had been tested and had failed.
Why the Search Gives Incomplete Answers
The semantic search finds the sections with the greatest content proximity to the question. Proximity, however, means similarity, not completeness. With relational domain knowledge, the two diverge.
An example from practice. In response to the question of whether an occupational therapist may dispense and bill assistive devices for her patients, the search finds the document on roles and permissions. It states that therapists may not issue prescriptions. The system derives a negative answer from this.
This answer is incomplete and therefore, as a result, wrong. Another document, in fact, sets out the condition under which the service is permitted: given a medical prescription and active delegation, the therapist may provide and bill the service. A conventional RAG chatbot thus turns a "Yes, but only if there is a medical prescription" into a "No."
The error is predictable once you understand how RAG works. The search rates sections by their proximity to the question, not by their substantive connection. The fact that an exception in a separate document belongs to a rule that was found is not recognizable to the process. Both appear as independent fragments, and the exception, which is linguistically less similar to the question, drops out of the result.
What is critical above all is the form the error takes. The answer is linguistically flawless, internally coherent, and appears complete. It is merely incomplete in substance. Where a piece of information leads to an action with legal or financial consequences, that is equivalent to a wrong answer. And because the answer sounds convincing, at first nobody notices.
For day-to-day operations, this has an immediate consequence. Anyone who has once received an answer that turned out to be incorrect will ask the service desk again next time. The assistant then provides no relief, but instead creates additional work.
The Approach: Graph RAG
The standard method rests on an assumption: that knowledge is a set of searchable text passages. For relational domain knowledge, that does not hold true. It is a network of rules, roles, exceptions, and conditions that depend on one another.
This is where graph-based methods come in, grouped under the term Graph RAG. The shared idea: alongside text-based knowledge, there is a knowledge graph that does not hold the content itself, but the relationships between the domain objects. Which role is authorized for which action, which rule restricts an action, which exception lifts a rule under which condition, which system state blocks which process.
This keeps the system from stopping at the rule it first finds. It follows the connections and pulls in the associated exceptions and conditions. In the example above, the node linking the therapist's role to the action in question is connected to the condition of active delegation and to the medical prescription. The answer is therefore not a flat refusal, but names the conditions under which the service is permitted.
Which Variant of Graph RAG
Graph RAG is not a single, uniform method, but a family of approaches that differ considerably.
At one end are fully graph-based methods, in which the graph itself forms the primary knowledge source, including automatically generated summaries of entire topic clusters. They are suited to broad, exploratory questions over very large data sets, but are costly to build and weaker in traceability.
At the other end are hybrid methods, in which the graph drives the search and the texts provide the explanation. The graph holds the verified structure, the vector search the running text along with its source.
For the case described here, we chose the hybrid approach, for three reasons. The questions are precise and rule-related, not exploratory. Every statement must be traceable to a specific source, which is non-negotiable in a regulated environment. And the graph remains limited to the relationships that are relevant to the domain, instead of mapping the entire data set, which keeps building and maintaining it manageable.
This decision is the actual core of such projects. Not choosing the most elaborate method, but the one that fits the structure of the knowledge and the requirements for traceability.
Building the Prototype
The following section describes the technical implementation. A prototype with two data stores was built for the feasibility study. A graph in Neo4j maps the rules and their relationships and is the starting point for every query. A vector store in Weaviate holds the running text and supplies the explanations for the objects found in the graph.
Building the Knowledge Base
During ingestion, content is read in from the content management system, SharePoint, and Confluence. Since some of the instructions are contained in screenshots, pure text extraction is not sufficient. The pages are processed multimodally while preserving the layout structure, so that action steps, tables, and warning notices remain recognizable as separate units.
Chunking is not based on fixed character lengths, but on the content's own structure, that is, on chapters, action steps, and rule blocks. A section that ends in the middle of a condition would otherwise lead to systematically incomplete answers.
The graph model maps the domain knowledge in a schema of eleven classes, including role, action, rule, exception, system state, module, error code, and insurance type. The relationships between these objects are the actual added value of the approach. Extracting them is at once the most labor-intensive step and the one that determines the quality of the overall system. Automatically generated relationships require expert review, since an incorrectly linked graph leads to answers that are just as convincingly worded as they are wrong.
Processing a Query
Answering takes place in four consecutive steps. The starting point is the graph, not the text search.
In the first step, a language model matches the question against the graph. It receives the schema and the existing entries and determines which domain objects the question concerns, that is, which roles, actions, rules, and exceptions are meant. For each match, it states how confident the assignment is. This way the system discloses what it has understood before it searches.
In the second step, the graph supplies the immediate neighborhood for these matches, that is, the connected nodes and edges. This is where the exceptions and conditions belonging to the rule that was found enter the context, whereas with pure text search they would have been left out.
In the third step, a separate query is sent to the vector store for each term matched in the graph. These queries are derived from the graph match, not from the question alone. They retrieve the explanatory running text along with its source for each object.
In the fourth step, the answer is synthesized. A model call receives the original question, the relationships from the graph, and the corresponding text passages with their sources. The instructions are tightly scoped: statements about rules, system states, and disruptions must be substantiated by the graph, textual statements are cited with the source file, and the underlying relationships are spelled out rather than just reflected in the result. The answer therefore contains not only the outcome, but also its reasoning.
Handling Uncertainty
If the data available for a question is insufficient, the system should not give an answer on a hunch. It flags the uncertainty and hands off to the service desk. An assistant that reliably answers a clearly defined subset of questions and passes the rest along in a controlled manner is preferable to a system that buys a higher hit rate at the cost of convincingly worded incorrect answers.
Traceability and Operations
Measurable evaluation of answer quality is part of running the system in production. A fixed catalog of real service desk questions with expert-verified answers serves as the benchmark against which every change is assessed for correctness, completeness, and relevance. Without such a benchmark, a gradual decline in quality would go unnoticed.
Every query is logged, including the sources drawn on and the answer generated. In a regulated environment, this traceability is not optional. Operations are set up on Swiss infrastructure, and the data is not used to train models.
Effort and Scope
A graph-based approach is not a free gain. The graph must be modeled, reviewed by experts, and kept up to date with every content change. That requires more infrastructure, more effort in preparation, and considerably more domain and technical knowledge than a pure vector search.
For a manageable, self-contained knowledge base, this effort bears no reasonable relation to the benefit. Anyone wanting to answer a limited number of clearly defined standard questions is served faster and more cheaply by the standard approach.
The extra effort is justified when at least one of the following conditions applies:
- A correct answer regularly requires combining multiple documents out of a large number of documents.
- The knowledge contains rules with exceptions, conditions, and role dependencies spread across different documents.
- An incorrect answer has legal or financial consequences.
If none of these conditions apply, the graph-based approach is the wrong choice.
Conclusion
Whether a standard approach suffices is decided not by the use case, but by the structure of the underlying data. The task of answering questions about a body of documentation is always phrased the same way. In practice, it ranges from trivial to demanding, depending on whether an answer lies in a single document or only emerges from the interplay of several.
A simple approach follows from this: the process begins with analyzing the data structure, not with choosing a tool. Only once it is clear how the knowledge is organized can one judge whether a standard method suffices or a graph-based approach is required.
The case described has been anonymized. The implementation was carried out as a prototype as part of a feasibility study. We are happy to help you assess your own use case.
