RAG Unpacked, Module 03: The Baseline You Should Beat Before Reaching for Vectors

This is the module I almost talked myself out of writing. Vector RAG is the famous version. Graph RAG is the structural alternative. What is left for a third module? The answer is the thing everybody skips on the way to embeddings: BM25, the keyword retriever that has been the production baseline for two decades, is older and simpler than vector retrieval, and on a surprising number of real workloads is strictly better.

The honest framing that emerged from actually building this is sharper than just keyword search. BM25 is the baseline you should measure before you pay for embeddings. If it solves 80% of your queries, adding vector retrieval is only worth the 20% it misses, minus the API costs and operational complexity. You can only compute that ratio after you have run the baseline. That is what this module is for.

The first question before reaching for an embedding model: what does BM25 already get right on your corpus?

What BM25 Actually Is

Three statistics about word counts, multiplied together. IDF rewards rare words. TF saturation counts repeated matches with diminishing returns. Length normalization means shorter documents win ties. That is the whole algorithm. The implementation in Python is roughly twenty lines. BM25 is from 1994 and it still works because most retrieval problems are not that hard: when the user's query and the relevant document share distinctive vocabulary, finding the document is mostly a counting problem.

What there is not: a neural network, a learned embedding, a transformer, or anything from the past decade of ML. No embedding API calls, no GPU, no index storage beyond a few KB of token statistics, and no embedding model to replace when a provider deprecates it. The free-ness is structural, not just free at small scale.

Why bm25s

Three libraries were on the table. rank-bm25 is pure Python and around 200 lines you can read end to end, but slow on real corpora. tantivy-py is Rust-backed and production-grade, but has a larger API surface. bm25s is about 10x faster than rank-bm25, has a single light dependency, and the full API fits on a cheat sheet: tokenize, BM25(), index, tokenize again for the query, retrieve. Five calls. Same kind of decision module 02 made when it picked Kuzu over Neo4j: keep the friction floor as low as possible without sacrificing portability of what you learn.

Three Notebooks: Crawl, Walk, Run

Crawl starts with a three-document toy corpus, then a 12-tip Python standard library corpus. The teaching beat that earns its place: the query 'binary search insert' surfaces tip_03 (bisect) with a clean score gap and zero scores behind it. That is the IDF reward made visible on only the second query. Then the teaser: swap one word in the next query, 'how do I sort a list of dictionaries,' and it fails because the relevant document says 'dicts' not 'dictionaries.' BM25 does not know they mean the same thing. That vocabulary-mismatch problem is what notebooks 2 and 3 are built around.

Walk covers the knobs: tokenization with stopwords on and off, stemming via PyStemmer (which collapses sort, sorted, sorting, and sorts to one token), and the k1 and b parameters with side-by-side rankings. The honest take: on a corpus this small, k1 shifts scores but rarely flips rankings. b barely moves anything because all 12 documents are similar in length. When BM25 is not producing the rankings you want, the problem is almost always upstream: your tokenization, your stopword list, or the vocabulary gap between your corpus and your queries. Not the hyperparameters. Defaults are good.

Run has three acts. Act I is the full vectorless RAG loop: tokenize, retrieve via BM25, augment a prompt, generate with gpt-4o-mini. Same four-step shape as module 01's vector RAG, one step swapped. The synonym-trap question from NB2 goes through the full loop and the LLM honestly reports it cannot answer from the context. The retrieval is the bottleneck, not the model.

The Head-to-Head: An Honest Result

Act II of NB3 loads module 01's Pinecone corpus and runs five engineered questions through both BM25 and vector retrieval. The questions were designed to show a spread: three rare-identifier queries where BM25 should win, one paraphrase query where vector should win, one well-worded question that should be a tie.

What actually happened: three agreements on the supposedly BM25-favored questions, two vector wins on the paraphrase queries, zero BM25-only wins. Not the spread I planned. On a 20-document corpus with distinctive identifiers, the embedding model converged on the same answers BM25 produced for keyword-shaped queries. The structural BM25 advantage on rare identifiers is real but it does not surface as disagreement when there are only 20 candidates. The gap shows up more clearly at scale, where vector has more semantically adjacent documents to confuse itself with.

That is a more honest result than what I had planned, and the notebook says so directly.

The RRF Surprise: Robustness, Not Quality

Act III combines BM25 and vector rankings with Reciprocal Rank Fusion: sum 1 divided by (k plus rank) across both lists, sort descending, take the top results. k=60 is the standard pick. Ten lines of code, no score normalization needed because RRF combines ranks, which are comparable across methods, not raw scores.

What I expected: hybrid would pick the right answer on every disagreement. What actually happened: hybrid picked the BM25 answer in both cases where vector was structurally better. In both cases the right document was vector's top result but BM25's fifth. The wrong-but-defensible BM25 first result was vector's third. RRF does not care which answer is correct. It sums ranks. The document that ranks high in both lists wins, even when the document that ranks first in just one list is the one the user actually wanted.

Hybrid is a robustness move, not a quality move. It stops you from falling off a cliff when one retriever fails. It does not reliably pick the better single-method answer.

To get back to best-per-query you layer a cross-encoder reranker on top of the fused list. That is module 05.

Five Takeaways

• BM25 is free. No embedding API, no GPU, no model deprecation risk. If your retrieval problem can be solved by keyword matching, BM25 solves it at zero marginal cost per query.

• Measure BM25 before you buy embeddings. The value of adding vector retrieval is only the queries it gets right that BM25 misses. You cannot compute that ratio until you have measured the baseline. Module 04 is where you do that measurement properly.

• The synonym trap is BM25's central failure mode. Stemming helps a little. Query expansion helps more. Vector search solves it natively. If users phrase questions in their own words rather than the vocabulary in your documents, you will feel this. If they search for error codes or API names, you probably will not.

• Hybrid is a robustness move, not a quality move. RRF gives you a system that does not catastrophically fail when one retriever is wrong. It does not guarantee you pick the best single-method answer on every query. For high-stakes retrieval you want a reranker on top of the fused list.

• Most retrieval problems are not ML problems. They are keyword problems where BM25 wins, vocabulary-mismatch problems where vectors win, or structural problems where graphs win (module 02). The interesting work is figuring out which problem you have.

Code: github.com/allllc/rag-unpacked   |   Module 03: 03-vectorless-rag

Next up: Module 04, Evaluating RAG. Faithfulness, context precision, answer relevance. How to actually measure which retriever wins on your traffic.

‍