An Overview of Information Retrieval Methods

Partly computed A handwritten illustration of a rewrite and of HyDE-style retrieval, standing in for a model's output. The searches run with them are real: BM25 (§3), and dense search (§5) with bge-small.

Stores

Nothing.

When a question arrives

A model, or rules, rewrites it in the documents' words, adds terms, splits it into parts, or drafts a hypothetical answer to search with; the search then runs on that.

The brain

The rewriting model's guess at how the documents would phrase the answer.

In the example

Rewritten as "sync token expired sign in again", the stale-login question finds the E104 passage first by BM25, which as asked never retrieved it. For the device question, a drafted answer moves accounts 1 from tenth to first by BM25: that is text expansion, the draft's words joining the search. Embedded instead, which is HyDE proper, the draft sits nearer the passage than the question did, though dense search already ranked it first. The draft's "ten" is wrong; the passage says five. A hypothetical answer is a search aid, never evidence: the answer comes from the passage.

Use it when

People's wording is far from the documents', or questions have several parts. It costs a model call before the search, and a rewrite can change the meaning, so log it and search with the original too.

Partly computed The searches' results are BM25's; the plan and the answer are illustrative.

Stores

Nothing new: it uses the search tools it is given.

When a question arrives

The model decides what to search for, reads the results, and searches again or answers, within a limit on rounds and tokens.

The brain

The language model's plan: which searches, in which words, and when it has enough.

In the example

Two searches find the two facts the relationship question needs. The same loop can follow a wrong lead: the libsodium search ranks share-links first, and a model that stops there answers wrongly.

Use it when

Questions need several steps or are worded vaguely. Each round costs a model call and time.

Computed with BM25 as the rank_bm25 library scores it.

Stores

An inverted index: for each word, the passages that contain it and how often. The same index serves Boolean retrieval, which returns exactly the passages that satisfy a condition built with AND, OR and NOT (an AND query keeps those holding all the words), and TF-IDF, the older weighting that BM25 refines.

When a question arrives

It is cut into the same words, and every passage holding any of them is scored.

The brain

BM25's score. Each shared word adds weight by how rare it is in the collection; more occurrences of a question word in a passage add more, with diminishing returns; and long passages are discounted, since they contain more words by chance.

In the example

Only errors 3 contains e104, so it alone scores. "What is E140?" goes wrong: among 23 short passages "is" is rare enough to carry weight, and E180's passage uses it twice. "Why does my phone say my login is stale?" shares no word with the E104 passage and never finds it.

Use it when

Questions carry exact identifiers, names or codes; you want no model to run; you want to see why something matched.

Computed with naver/splade-cocondenser-ensembledistil. Terms such as "ex" and "##pire" are the model's word pieces.

Stores

For each passage, a weight for each of a few dozen vocabulary terms the model chose, including terms the passage doesn't contain: about 96 per passage here.

When a question arrives

The model weights the question's terms the same way, and a passage scores the sum of the products of the terms both have. Most weights are zero, so an inverted index still serves it.

The brain

The trained model that decides which terms describe a text and how much, adding related ones.

In the example

For "Cached results stay valid for 15 minutes…" the model added long, period and years; the question "How long before saved results expire?" carries long, period and years too, so they meet where BM25 found little. "years" shows the additions are learned guesses.

Use it when

You want keyword-style indexes and visible matching terms, with part of the vocabulary gap closed.

Computed with BAAI/bge-small-en-v1.5. The map is a two-dimensional projection (PCA) of the real 384-number vectors, so distances in it are approximate.

Stores

One vector per passage, 384 numbers here, from an embedding model.

When a question arrives

The question is embedded too (this model expects a short instruction before a question), and the passages whose vectors are nearest by cosine are returned.

The brain

The embedding model, trained so that passages relevant to a question get vectors near the question's. A question and its answer don't mean the same thing; the model learns to place them close anyway. Most of the deciding happens inside the model, before any comparison.

In the example

The expiry question lands beside caching 2 and releases 3, the two passages that answer it. "Why does my phone say my login is stale?", which BM25 missed, ranks the E104 passage second, behind the suspended-account passage.

Use it when

People's words differ from the documents'; with a multilingual model, across languages. Each question needs an embedding computed, which a small model can do locally, with no paid call.

Computed with answerdotai/answerai-colbert-small-v1; the values are each question token's best similarity in the passage.

Stores

A vector for every token of every passage: about 25 vectors of 96 numbers per passage here, about six times as many vector values as one dense vector, before compression and index overhead.

When a question arrives

Each question token, embedded in its context, finds its most similar token in a passage; the passage scores the sum of those best matches (MaxSim).

The brain

Contextual token vectors and the sum of best matches: a passage isn't squeezed into one point, so the match is kept word by word.

In the example

"saved" found "stored", "results" found "results", "before" found "after", and "long" found "for", as in "for 15 minutes". caching 2 scored 30.75 and releases 3 30.67: the model also adds padding tokens to every question, so scores sit close together and small differences decide.

Use it when

Precision matters and you can afford a larger index, or as a reranker over a shortlist.

Computed with SQLite. The table is the app's error list, a source the help pages don't contain.

Stores

Rows with explicit fields: here each error's code, meaning, the version that added it, and whether it stops syncing. The version is a plain number to keep the example short. Real version numbers need a version-aware comparison: 4.10 comes after 4.9, while the number 4.10 equals 4.1.

When a question arrives

It becomes conditions, joins and counts, written by a person, a form, or a model writing SQL, and the database returns every row that meets them.

The brain

The query's logic over the fields. Nothing is ranked by relevance: a row meets the conditions or it doesn't.

In the example

Two rows match, and a count returns 3. Top-k relevance retrieval, which returns the few best-scoring passages, can't establish that a list is complete; a text index can still match exhaustively and count, as a Boolean query over an inverted index does. Metadata filters bring the same idea into search: keep only passages for version 4 or later, then rank the rest.

Use it when

Questions involve dates, numbers, exact lists or counts, and the facts live in fields. A model writing SQL needs a read-only connection and limits.

Computed as a join of two link types over nine stored facts. Result: e2e-encrypt.

Stores

Facts as typed links between things: (desktop app, loads, e2e-encrypt), (e2e-encrypt, depends on, libsodium).

When a question arrives

It becomes a pattern of links, and every path that matches the pattern is returned.

The brain

Pattern matching over the stored links. The answer is exact and complete relative to what the graph holds and how the pattern is written: a missing or wrong link gives a missing or wrong answer.

In the example

The join returns e2e-encrypt and excludes share-links. Text search finds the two passages that hold the facts, plugins 1 and plugins 4, but leaves the joining to whoever reads them.

Use it when

Questions chain facts, need complete lists, or need the path as an explanation, and someone keeps the graph current.

Illustrative Not run: it needs a model trained on the collection.

Stores

The collection inside a sequence-to-sequence model's weights, each document known by an identifier the model learned to write.

When a question arrives

The model generates identifiers token by token; the most likely ones are the results.

The brain

The trained weights: there is no separate index to look in.

In the example

Not run.

Use it when

Mostly research, and small or structured sets such as entities or products. It has been studied at the scale of millions of passages, 8.8 million in Pradeep and colleagues' 2023 study, with open problems in quality, compute, and adding documents without retraining.

Computed BM25, dense by bge-small, and fusion as the formula states.

Stores

Nothing of its own: it combines the lists of two or more first-stage retrievers.

When a question arrives

Each retriever returns its top results; each result scores 1 / (60 + its rank) in every list it appears in, and the sums decide.

The brain

A vote by position. Agreement between lists counts for more than a high place in one, and the scores themselves are ignored.

In the example

Here fusion hurt: the answer was second by meaning and fell to fourth, because BM25's list agreed with dense on three wrong passages. With a weak keyword list that can happen; where questions carry identifiers, it often goes the other way. Measure it on your own questions.

Use it when

You have two retrievers that fail differently. A weighted sum of normalized scores, tuned on judged questions, can do better than this vote.

Computed with cross-encoder/ms-marco-MiniLM-L6-v2 over the fused top 10.

Stores

No index of its own: it receives the candidates' text at question time.

When a question arrives

The reranker reads the question together with each candidate and gives each pair a score; the candidates are reordered by it.

The brain

A model that reads both texts at once, so it can tell a passage that answers the question from one that only shares its words.

In the example

For "Which version added end-to-end encryption, and which library does it use?", the fused top three held the release note for version 4.2 but not plugins 4, the passage that names libsodium, which fusion had placed sixth. The reranker moved it to second, so the top three held both facts the answer needs. For "Why does my phone say my login is stale?" it assigned low raw scores to all ten candidates, between −11.1 and −11.5, although the E104 passage was among them; it ranked that passage third.

Use it when

You can afford to run the model over every candidate for every question. It can only promote what the first stage found.

Partly computed The passages are the reranker's real top two; the answer is illustrative.

Stores

Whatever the retrievers store.

When a question arrives

The best passages go into the prompt, numbered, with an instruction to answer from them and cite them; a language model writes the answer.

The brain

The language model reading what it was given, together with everything it learned in training. Instructions steer it toward the passages; they don't guarantee it uses only them.

In the example

Both passages that hold the answer reached the prompt. Had retrieval missed them, the right reply would be that the pages don't say; a model may guess instead.

Use it when

People want an answer in words with sources they can check. Check the citations: a citation can point to a passage that doesn't say what the sentence claims.

Illustrative A deliberately bad cut of the caching page, to show the failure.

Stores

The documents as cut into chunks, sometimes with each chunk's parent section or a summary of it.

When a question arrives

Retrieval returns chunks. Small-to-big retrieval matches small chunks and returns the section around them; hierarchical retrieval finds the right section or summary first, then searches inside it.

The brain

The chunking rule. It decides what can be matched and what arrives with a match.

In the example

Cut this way, the chunk that holds "15 minutes" never says what is replaced. The real pages were cut at paragraphs, which kept each fact with its subject; a heading dropped in cutting, or a filter that excluded release 4.3's notes, would still hide part of the answer.

Use it when

Always: even a strong retriever can't return an answer that was split from its explanation, stripped of its heading, or filtered out. Measure retrieval with your own cutting.

Illustrative Not run. One graph-based RAG design, Microsoft Research's GraphRAG (Edge and colleagues, 2024); other designs differ.

Stores

A graph a model extracted from the documents, the communities of closely linked entities in it, and a model-written summary of each community.

When a question arrives

Local search starts at the entities the question names and gathers their neighbours and source passages; global search answers collection-wide questions, such as "what are the main themes?", from the summaries.

The brain

The extracted graph and its summaries, read by a language model that writes the answer.

In the example

Not run. It differs from §8: there the graph was written by hand and the answer is an exact path; here the graph is extracted by a model and the answer is generated.

Use it when

Questions span a whole collection rather than one passage, and the model calls that building takes, for extraction, retries and summaries, are affordable.

Illustrative weights and threshold; the Jaro–Winkler similarity of jon and john, 0.93, is computed.

Stores

Records with fields, from one source or several.

When records arrive

Candidate pairs are compared field by field. An agreement adds weight by how unlikely it is by chance; a disagreement subtracts (Fellegi–Sunter).

The brain

The field weights and a threshold. The threshold is a decision rule, not proof: above it the system predicts a match, with uncertainty, and close cases go to a person.

In the example

With a fuzzy name comparison the pair scores 19.63 and is predicted to match. Compared exactly, the given names count against the pair, −4.31 instead of +6.57, and 8.75 sends it to review.

Use it when

Merging sources, removing duplicates, and keeping a graph's nodes from splitting one thing in two.

Illustrative Positions made up to show the mechanism.

Stores

Training pairs: a question and a passage that answers it, with negatives selected for each question.

When it trains

Each question is pulled toward its passage and pushed from its selected negatives: the other passages in the batch, or hard negatives mined with BM25. A selected negative can turn out to be a valid answer too, which teaches the model something false.

The brain

The training objective (contrastive, such as InfoNCE) that shapes the vectors dense retrieval compares.

In the example

Not run. The off-the-shelf bge-small model already placed the expiry question beside its answers (§5).

Use it when

Off-the-shelf embeddings don't fit your field. Changing the model that encodes documents usually means encoding every passage again; an adapter applied to questions alone avoids that.

Illustrative

Stores

Whatever training put in the model's weights.

When a question arrives

The model answers directly.

The brain

The weights.

In the example

Acme Sync was made up for this page. A model without access to these invented pages has no basis for knowing their contents; it could only guess.

Use it when

General knowledge is enough. It leaves no retrieved evidence trail: a model can write citations, but they may not be accurate.

Diagram

The brain

A vision-language model that embeds an image of each page as many patch vectors, matched against the question's tokens by late interaction, as in §6.

What it adds

Tables, charts, diagrams and layout are searched as they appear on the page, with no text-extraction step to lose them. Every example on this page retrieves text.

Where it stands

Faysse and colleagues introduced ColPali in 2024, with a benchmark of visually rich documents, ViDoRe, on which it largely outperformed pipelines that extract the text first.

Diagram

The brain

Working out what a question needs, such as the principle, the theorem or the intermediate fact, before deciding which documents matter.

What it adds

Similarity finds documents that resemble the question, and some relevant documents don't: a coding question can be answered by a solved problem that uses the same algorithm in other words. The agent of §2 changes what is searched for; here the system has to recognize relevance that resemblance doesn't show.

Where it stands

BRIGHT (Su and colleagues, 2024) collects 1,384 such questions from fields including economics, psychology, mathematics and coding. The model leading the MTEB embedding leaderboard at the time scored 18.3 out of 100 on it (nDCG@10, a measure of ranking quality), and having a language model write out its reasoning about the question first improved retrieval by up to 12.2 points.

Diagram

The brain

A language model given the question and several candidates together, asked to write their order.

What it adds

The candidates are compared with each other, where the cross-encoder of §11 scores each pair on its own. A longer list goes through in overlapping windows.

Where it stands

Sun and colleagues (2023) found properly instructed language models "competitive, even superior" to supervised rerankers on standard benchmarks, and distilled the ability into smaller models. The order the candidates are given in can change the result (Tang and colleagues, 2023).