What Is Reranking in AI?

Reranking is a second pass over search results that reorders them by how well each one actually answers your query. A retrieval step pulls back a rough shortlist, usually a few dozen candidates, and the reranker looks at the query and each candidate together, scores the real match, and pushes the best ones to the top. It runs after retrieval and before the AI model reads anything, so the model sees the five documents that matter instead of the fifty that were merely close.

If you have built anything with retrieval augmented generation, reranking is the cheapest quality upgrade available to you. It does not require training a model, and it usually takes an afternoon to wire in. If you are still deciding between retrieval approaches, our comparison of RAG, fine-tuning, and long context is a better starting point than jumping straight to reranking.

Where reranking fits in a RAG pipeline

A typical retrieval pipeline has three stages. Reranking is the middle one that most first builds skip:

1. Retrieve. A vector search over your embeddings returns the top 30 to 50 chunks that are numerically close to the query.

2. Rerank. A reranker model scores each of those chunks against the query and reorders them by true relevance.

3. Read. You keep the top 3 to 8 reranked chunks and pass only those into the model as context.

The retrieval step is fast but blunt. The reranking step is slower per document but far more accurate, which is why you only run it on the shortlist and not on your whole database.

Why the first pass is not enough

First-pass retrieval uses a bi-encoder. It turns your query into one vector and every document into its own vector ahead of time, then compares them by distance. That is fast because the document vectors are precomputed, but the query and the document never actually meet. A reranker uses a cross-encoder. It feeds the query and one document into the model together, so the model can weigh the exact words against each other. That is slower, but it catches matches the vector distance misses.

A concrete before and after

Say your app answers questions about a company handbook and someone asks, can I expense a client dinner. Plain vector search returns the general expenses policy, the per-diem travel section, and a page about corporate cards, because all three are near the word expense. The clause that actually answers the question, a two-line rule about client entertainment under meals, sits in eighth place because it never uses the word expense at all.

A reranker reads the query against each candidate and recognises that the client entertainment clause is the direct answer, so it moves to first place. Same retrieved set, better order, and now the model quotes the right rule instead of guessing from the general policy.

When reranking is worth the latency

Reranking adds tens to a few hundred milliseconds per query. That is worth it more often than not, but not always. Reach for it when:

• Your answers are wrong or vague and you can trace it back to the model getting the wrong chunks.

• Your documents overlap heavily, so many chunks look similar to a vector search.

• You retrieve a wide net (30 or more) to improve recall and need something to sort the noise.

• Correctness matters more than shaving 100 milliseconds, which covers most support, legal, and internal tools.

You can skip it when your corpus is small and clean, when queries map one-to-one to obvious documents, or when the app is latency-critical and the current answers are already good. Do not add a reranker to a pipeline that is not measurably failing. It is a fix, not a default.

How to add reranking without training anything

You have two routes and neither needs a data scientist. The first is a hosted rerank API, where you send the query and the candidate list and get back scores; Cohere's Rerank documentation is a clear reference for the request and response shape. The second is an open cross-encoder you run yourself, such as a bge-reranker model, which keeps everything in your own infrastructure. If you are wiring this into a product, it usually lives right behind the feature that adds search to your app. The call looks the same either way:

# pseudo-code for the rerank step
candidates = vector_search(query, top_k=40)      # fast, blunt
scored = rerank(query, [c.text for c in candidates])  # slow, accurate
top = sort_by_score(scored)[:6]                  # keep the best few
answer = model.generate(query, context=top)

Start by retrieving more than you think you need, rerank down to a handful, and measure whether answer quality improves. If it does not, your problem is upstream in chunking or retrieval, not ranking.

Frequently asked questions

Does reranking replace embeddings?

No. Reranking works on top of embedding-based retrieval. You still need the fast first pass to build the shortlist. The reranker only reorders what retrieval hands it, so if the right document was never retrieved, no reranker can save you.

How many documents should I rerank?

Retrieve 20 to 50 and rerank down to 3 to 8. Reranking more than about 50 gets slow without helping, and reranking fewer than 10 rarely changes the order enough to matter.

Does reranking increase cost?

A little. Hosted rerankers charge per query or per document scored, and self-hosted ones use compute. Because you only rerank a shortlist, the cost is small next to the model call that follows, and it often lets you send fewer chunks to the model, which saves tokens.

Is this the same as ranking in a recommender system?

The idea is related, ordering candidates by relevance, but the tools differ. In AI retrieval a reranker is usually a cross-encoder language model scoring text against a query, not a recommender trained on click data.