The Lying Reranker

Chapter 3 — The Lying Reranker

In the technical jargon of RAG, there's a key, unglamorous, and often untold figure: the reranker. Imagine company search like the entrance to a nightclub. The vector database is the bouncer at the door: it quickly scans the crowd and lets in thirty candidates who, at a glance, seem like what you're looking for. The reranker is the inside doorman: it takes those thirty, looks at them calmly, selects five — the ones the LLM will actually read to build the answer — and sends the other twenty-five back. It's a small but enormous function: if it fails, everyone fails. If the doorman can't read faces, the club fills up with the wrong people and no one will ever understand why the night went wrong.

In 2025, several cloud services appeared on the market offering rerankers as APIs. You send a query and a list of documents, and they return scores. Convenient, scalable, no GPUs to buy, no models to download. A famous European model provider — without naming names, but you know which names we're avoiding — exposed a promising model from the Qwen family, with four billion parameters, labeled "Reranker". The price was reasonable, the latency acceptable. Anyone, at that historical moment, would have chosen it. Many did choose it.

Unless the scores were wrong. Not "slightly" wrong, as can happen with any model: structurally wrong. Documents blatantly relevant were getting 0.2, off-topic documents were getting 0.8. The first suspicions, as always in these cases, were directed at the usual suspects: the embedding model, the chunking, the query formulation, the document preprocessing. Weeks of investigations down the wrong path. Only by systematically comparing the responses of the cloud reranker with those of a local reference reranker — the same input, the same documents, scores side-by-side on an Excel sheet — did the uncomfortable truth emerge: the model exposed via API was broken. Perhaps a deployment error, perhaps a wrong version uploaded by mistake, perhaps a bug in the serialization of the scores. The provider never formally admitted it. The problem, however, simply disappeared one day, after a silent update and no release notes.

The point of this story isn't "cloud services make mistakes" — everyone makes mistakes, even local models make mistakes. The point is more subtle: in a serious RAG, the reranker is a piece you need to be able to look inside. If it's a paid black box, and if its outputs are numbers that seem plausible even when they are random — and reranker scores always seem plausible, because they are numbers between zero and one with some decimals — you have no way of understanding what's wrong with your system. And since the reranker is "towards the end" of the pipeline, its error contaminates every evaluation upstream: the search seems slow, the query wrong, the embeddings poor. In reality, it's the bouncer who can't read faces, and you're questioning the glass in the door.

The counter-current choice of some teams, in recent months — instead of going towards more cloud, returning back — has been to bring the reranker in-house. An open source model from the BGE family, not gigantic, running on local GPUs (even on Apple Silicon, with some caution regarding drivers). More management work, it’s true. But the ability to run controlled experiments, to understand when it fails, to compare versions, to keep a history. And — not least — to avoid paying the API a fraction of a cent for each single user search. A fraction that, multiplied by tens of thousands of queries per month, quickly stops being a fraction and becomes a budget item.

When a component is so critical that its malfunction corrupts your every ability to measure everything else, then delegating it to a black box is not efficiency. It’s an act of faith. And acts of faith, in production, are paid for with compound interest.