Beyond Basic RAG: Building a More Reliable Retrieval Pipeline With Hybrid Search

Retrieval-Augmented Generation (RAG) is a practical way to connect large language models to private or domain-specific information.

Instead of asking an LLM to answer only from what it learned during training, a Retrieval-Augmented Generation system retrieves relevant information at query time and provides that context to the model before it generates a response.

The basic RAG pipeline is relatively simple.

A user asks a question, the system converts that question into an embedding, retrieves semantically similar documents using vector search, and sends the most relevant results to an LLM.

That can work well in prototypes.

But production systems tend to expose a harder problem:

Not every question is well suited to semantic vector search.

A developer may search for an exact function name. A support user may enter an error code. A legal professional may need a particular clause number. Someone else may ask a vague question without enough context for useful retrieval.

In each case, the underlying model may be perfectly capable of answering the question if the retrieval layer can first find the right information.

This is where hybrid search becomes useful. Rather than relying on a single retrieval method, hybrid search combines semantic vector retrieval with lexical or keyword-based search. PostgreSQL provides built-in full-text search capabilities, while pgvector adds vector similarity search to Postgres.

As part of our AI development work at IT Path Solutions, we explored a RAG architecture designed around this problem.

Rather than treating retrieval as one vector lookup, we built it as a multi-stage workflow combining:

• Query rewriting
• Semantic vector search
• Exact-term search
• Rank fusion
• Cross-encoder reranking
• Relevance checks before generation

Internally, we refer to this approach as Graph Vectoring because the retrieval process is implemented as a graph of individual decision and retrieval steps rather than a fixed search operation.

The important idea, however, is not the name.

It is treating RAG retrieval itself as an engineered pipeline.

Where Basic Vector RAG Can Struggle

Vector search is designed to find information based on semantic similarity.

That makes it useful for questions where the exact wording used by the user may differ substantially from the wording in the source material.

For example:

“How can I prevent an API from receiving too many requests?”

A vector search system may successfully connect that question with documentation discussing rate limiting, even if the user never uses those exact words.

But semantic similarity becomes less reliable when the exact term itself matters.

Consider queries involving:

• Function names
• API signatures
• Product SKUs
• Error codes
• Legal clause numbers
• Configuration parameters
• Specific identifiers

A document may contain exactly the information the user needs while still receiving a weaker embedding-similarity score than a conceptually related document.

The opposite problem occurs when the user’s query is vague.

A question such as:

“Why isn’t this working?”

may contain too little information for a useful retrieval query.

If irrelevant context is retrieved and passed to the LLM, the generation step may still produce a fluent answer. But fluency does not guarantee that the answer is grounded in the right source.

That makes retrieval quality one of the important engineering considerations when moving a RAG application from prototype to production.

Our Approach: Treat the RAG Pipeline as a Workflow

Instead of sending the user’s original query directly to a vector database, our implementation passes it through several stages.

The architecture can be simplified as:

User Query → Query Rewrite → Hybrid Retrieval → Rank Fusion → Reranking → Relevance Check → LLM Generation

If retrieval quality is not strong enough, the workflow can return to the query-rewriting stage and attempt retrieval again.

This changes the RAG pipeline from a single lookup into an observable decision process.

If you are deciding where tools such as LangChain, LangGraph, and RAG fit within an AI product, our architectural guide to LangChain, LangGraph, and RAG covers those roles in more detail.

Step 1: Rewrite the Query Before Searching

Real users rarely write queries specifically for a retrieval engine.

Questions may be conversational, incomplete, ambiguous, overly long, or missing terminology that appears in the underlying knowledge base.

The first node therefore uses an LLM to reformulate the input into a query better suited to retrieval.

Importantly, the original query is preserved.

The rewritten query exists to improve search, not to change what the user intended to ask.

For example, a user might write:

“Why does checkout keep failing after login?”

Depending on the available context, the rewritten query could surface clearer concepts or terms that are more useful to both semantic and lexical search.

This means the user does not have to understand how the underlying retrieval system works or learn how to write an optimized search prompt.

Step 2: Run Vector Search and Keyword Search Together

The rewritten query is sent through two retrieval methods in parallel.

Dense Vector Search

PgVector handles semantic retrieval using vector embeddings.

This works particularly well when the query and the relevant source discuss the same concept using different language.

For example:

“How do I stop users from making too many API requests?”

could still retrieve documentation about:

“Implementing rate limiting.”

Lexical Search

PostgreSQL full-text search handles keyword-oriented retrieval.

This becomes valuable when exact terminology matters, such as:

• NullPointerException
• getUserById()
• ERR_CONNECTION_REFUSED
• SKU-10428
• A contract clause number

Semantic search and lexical search therefore solve different parts of the retrieval problem.

Rather than choosing one, this hybrid search RAG architecture uses both.

Step 3: Combine the Search Results

Running two retrieval systems creates another problem: each produces its own ranked result list.

The system therefore uses Reciprocal Rank Fusion (RRF) to merge the dense and lexical results into one candidate ranking.

RRF allows highly ranked documents from either retrieval method to rise within the combined result set without requiring the system to directly compare two different kinds of search scores.

The result is a broader candidate pool containing documents discovered through semantic similarity, exact-term relevance, or both.

Step 4: Add RAG Reranking

The candidate set then passes through a cross-encoder reranker.

This stage performs a more direct relevance comparison between the query and each candidate document.

Vector search is useful for quickly identifying potentially relevant documents from a much larger corpus.

The RAG reranking stage performs the more focused task of deciding:

Which of these candidates is actually most relevant to this particular query?

The highest-ranking documents can then be selected as context for the LLM.

This two-stage approach allows the retrieval layer to use relatively broad search first and more precise relevance evaluation afterward.

For production software teams, these kinds of architectural decisions often sit alongside broader application concerns such as observability, integrations, performance, and scale. That is also why RAG systems frequently become part of a wider custom software development effort rather than remaining an isolated AI proof of concept.

Step 5: Decide Whether the Retrieved Context Is Good Enough

One of the more useful properties of implementing the pipeline as a graph is that generation does not have to happen automatically after retrieval.

Before sending the retrieved context to the LLM, the workflow can evaluate whether the top results meet a relevance threshold.

If they do, the system continues to generation.

If they do not, the graph can return to the query-rewriting node and attempt another retrieval pass.

The workflow becomes:

Retrieve → evaluate → proceed or retry

rather than:

Retrieve → generate regardless of quality

We refer to this as self-correcting behavior, although the important distinction is that the system is not “correcting” the LLM itself. It is giving the retrieval process another opportunity when the first search produces weak results.

Why Build the RAG Architecture as a Graph?

The individual retrieval techniques are useful, but the graph-based RAG architecture provides another advantage: each stage remains separate and observable.

In our implementation, every LangGraph node reads from and writes to shared typed state.

That makes it possible to inspect:

• The original user query
• The rewritten query
• Dense search results
• Lexical search results
• Fused rankings
• Reranker scores
• The final context selected for generation

If an answer is poor, the team can inspect the retrieval path and identify where the problem occurred.

Was the query rewritten badly?

Did vector search retrieve the wrong documents?

Did lexical search miss an exact term?

Did the reranker prioritize the wrong result?

Did the system proceed despite weak retrieval confidence?

This level of observability becomes increasingly useful when debugging AI systems because a poor generated answer does not necessarily mean the LLM itself is the problem.

Why We Used Postgres for Hybrid Search

Another architectural decision was to use PgVector and PostgreSQL full-text search together.

PgVector provides dense vector retrieval, while PostgreSQL already provides lexical full-text search.

That means both retrieval methods can operate against the same underlying database rather than requiring a separate search platform purely to introduce keyword retrieval.

This does not mean Postgres is automatically the right search architecture for every RAG system.

Large-scale search workloads or applications with different search requirements may justify dedicated infrastructure.

But for teams already using PostgreSQL, combining PgVector with full-text search can provide a relatively straightforward path toward hybrid retrieval without immediately adding another search system to operate.

Where This RAG Architecture Can Be Useful

The need for hybrid retrieval depends heavily on the type of information users are searching.

Technical Documentation and Developer Assistants

Developer questions often combine conceptual and exact-match retrieval.

A developer might ask:

“How should authentication tokens be refreshed?”

and later search for:

refreshAccessToken()

The first benefits from semantic retrieval.

The second benefits from lexical matching.

A hybrid RAG pipeline can support both within the same knowledge system.

Customer Support Systems

Support queries are frequently incomplete or written using terminology that differs from internal documentation.

Query rewriting can clarify the search intent, while hybrid retrieval helps locate relevant manuals, troubleshooting documentation, or knowledge-base content before the LLM generates a response.

Legal and Compliance Knowledge Systems

Legal retrieval frequently requires both conceptual similarity and exact wording.

A user may want material related to a particular concept while also needing a specific clause, phrase, or reference.

Using semantic and lexical retrieval together gives the system more than one way to locate that information.

These patterns are particularly relevant for teams building AI assistants and agentic workflows that need dependable access to business data. IT Path Solutions also works on agentic AI development for products where orchestration, context handling, and bounded decision-making matter alongside retrieval itself.

When Is Basic RAG Still Enough?

Not every RAG application needs this architecture.

A simple vector-search pipeline may be completely sufficient when:

• The knowledge base is relatively small
• Queries are straightforward
• Exact identifiers rarely matter
• Retrieval quality is already consistently good
• Additional pipeline complexity would provide little practical benefit

Every additional stage introduces engineering and operational considerations.

Query rewriting adds an LLM call.

Reranking adds another inference step.

Retry logic can increase latency.

More stages also mean more components to observe and evaluate.

The question therefore should not be:

How many retrieval techniques can we add?

It should be:

What retrieval problems does the application actually have?

Architecture should follow those requirements.

What Teams Should Evaluate in a Production RAG System

Once a RAG application moves beyond a proof of concept, evaluating only the LLM is rarely enough.

Teams should also examine the behavior of the retrieval pipeline itself.

Some useful questions include:

Can the system find exact terminology?

If users regularly search for identifiers, codes, product names, or technical strings, vector similarity alone may not be enough.

What happens when the user’s question is vague?

The system should have a strategy for poor or incomplete queries rather than blindly generating from weak context.

Can retrieval quality be inspected?

Teams need enough visibility into search results and rankings to determine why an incorrect answer occurred.

What happens when retrieval confidence is low?

Depending on the application, the system may retry, ask the user for clarification, decline to answer, or use another retrieval strategy.

How much latency does each stage introduce?

Better retrieval only helps if the overall experience still meets the application’s performance requirements.

These decisions are part of production RAG architecture, not simply prompt engineering.

What We Learned From Building the RAG Pipeline

The biggest takeaway from this work was that improving a RAG application is not always about using a larger or more capable language model.

Sometimes the limiting factor sits earlier in the pipeline.

If the right source never reaches the model, generation quality can only go so far.

Combining vector search with exact-match retrieval gave the system two different ways to find relevant information. Reranking added another relevance filter, while the graph architecture allowed the workflow to retry when the initial retrieval results were not strong enough.

Just as importantly, separating the process into observable stages made the system easier to inspect and debug.

For teams building AI-powered search, assistants, internal knowledge systems, or support applications, that distinction matters.

A dependable RAG system is not simply an LLM + vector database.

It is a retrieval and generation architecture designed around the kinds of questions real users actually ask.

At IT Path Solutions, experiments like this help our AI engineering teams understand those architectural trade-offs before applying them to production systems.

The goal is not to make every RAG pipeline more complex.

It is to make it as sophisticated as the problem requires, and no more.