Building Production RAG: Combining BM25, Dense Embeddings & Cross-Encoder Rerankers

A step-by-step engineering blueprint to eliminate hallucinations and achieve 95%+ retrieval accuracy in production knowledge bases.

Prerequisites

Python 3.10+
Basic vector database concepts
Familiarity with LangChain or LlamaIndex

Step 1: Step 1: The Failure Modes of Naive Cosine Similarity

Naive vector search treats all queries as semantic concepts. If a user asks for exact part number "SKU-9941-X", dense embeddings often retrieve similar-sounding text rather than the exact alphanumeric match. We solve this with hybrid retrieval.

Step 2: Step 2: Implementing Reciprocal Rank Fusion (RRF)

Combine keyword search rankings with vector distance using Reciprocal Rank Fusion formula: RRF_Score = sum(1 / (k + rank_i)).

Step 3: Step 3: Cross-Encoder Reranking for Precision

Take the top 25 fused candidates and pass both the original query and candidate text through a cross-encoder model (e.g. bge-reranker-large) to score true pairwise entailment.