The Inherited Hallucination

Chapter 4 — The Inherited Hallucination

This is perhaps the most instructive story of the entire investigation, because it touches on a theme that tutorials ignore and that instead causes more damage than all the others combined: the quality of the data that enters.

An American bicycle company — let's call it that out of respect — had a huge photographic archive, thousands of product images, events, customers on bikes, promotional trips. It was the company's visual heritage, accumulated over fifteen years of campaigns. In a moment of foresight, someone had used an automatic tagging service to add, to each photo, a list of "objects present": "bike, helmet, road, mountain, person". Metadata useful, in theory, for future visual search. Stored next to the photos, forgotten for years, then recovered and ingested into the new company RAG with the natural conviction that "more metadata is better".

After ingestion, image searches yielded surreal results. You searched for "bikes in the city" and got a photo of a lake. You searched for "female model with helmet" and got a vintage car. You searched for "mountain" and got, indiscriminately, anything — a random sample from the entire archive, as if the system had drawn lots. The team initially pointed the finger at the visual embedding model. Then at the reranker (see previous chapter, to understand why this track was particularly seductive). Then at the quality of the descriptions generated by the AI.

The real investigation, conducted this time with a direct SQL query to the database, revealed something grotesque. Of the 3997 catalog photos, all, absolutely all, had the exact same list of objects. A list of nineteen items — "car, motorcycle, lake, sea, mountain, bicycle, helmet, road, city, person..." — which had no relation to the content of the individual photo. It had been imported, years ago, from another website, as a set of generic tags to fill a mandatory field in the CMS. No one in the company remembered it anymore. The photos had passed from system to system, carrying those ghost labels with them, plausible enough not to raise suspicion, poisonous enough to corrupt any search based on them. The search engine, diligent as it was, used them as a symptom of content. And returned any photo for any query, with a certain degree of hallucination guaranteed by the laws of nature.

The hard part, the one that hurts when you understand it, is that in a RAG, "noisy" data doesn't make noise that can be heard. It doesn't slow down the system, it doesn't generate errors, it doesn't trigger alarms. It simply silently erodes the quality of the results. And when the downstream language model receives five documents, four of which are off-topic, it doesn't tell you "these are irrelevant, ask me something else": it diligently constructs a response that mixes them, and the response always sounds plausible. Always. The final hallucination — that invented answer we complain about — doesn't originate from the model, as told in the dominant narrative. It originates upstream, from the data. The model is just the one that packages it into a grammatically impeccable sentence. But the blame lies elsewhere, further back, in what you fed the system months before you asked it anything.

This observation has practical consequences that cost money. The first task for anyone building a serious enterprise RAG, before writing a single line of code, is to perform an autopsy of their data. Where does it come from? Who has touched it? Which fields are observed and which are fossilized legacies of previous systems? Which metadata have meaning today and which had it in 2017 and no one has bothered to clean up? It’s a less-than-glamorous, very archival job, completely absent from tutorials, often perceived as "non-technical" and therefore snubbed by development teams. But without that initial hygiene, any brilliant architecture you put upstream simply becomes a very powerful amplifier of very old garbage.

In the case of the American company, the solution was brutal and correct: empty the "objects" field, rebuild the metadata from scratch with a modern vision model applied to each individual photo, and track, this time, when and how each metadata was produced. Tedious work. Result: search finally started to work. Not because they changed the model. Because they cleaned the basement.