The Engine That Had Six Poles — The Myth of Ten Lines of Code

Six-part investigation into the reasons why the document chatbot you were sold continues to invent things — and the different direction that, at the beginning of 2026, someone began to take.

Six-part investigation into the reasons why the document chatbot you were sold continues to invent things — and the different direction that, at the beginning of 2026, someone began to take.

Intro — The Engine That Had Eight Poles (But Had Six)

Milan, March 2026. A manager opens the internal company chatbot, the kind assembled "in two weeks" by the IT department with great enthusiasm and little distrust. A very simple question: "How many poles does the CMP40M engine have?" The chatbot replies with all the confidence that large language models know how to exhibit: "The CMP40M engine has eight poles."

Wrong. It has six. The correct answer wasn't in the 422-page catalog that someone, months before, had confidently poured into a vector database, convinced that from then on the system "would know everything". That data simply wasn't in the catalog. It was in a PDF attached to an email from SEW technicians, archived in a folder that no one had bothered to index.

This scene, with variations in sector and dialect, is repeated in hundreds of Italian offices. The promise was simple and seductive: give all your company documents to your AI assistant, and it will be able to answer any question about them. The reality is much more prosaic: the chatbot reads but doesn't understand, searches but doesn't find, and when it doesn't find — instead of saying so — it invents. It invents fluently, with perfect grammar, with plausible numbers. Which is much worse than not answering.

Between the end of 2024 and the beginning of 2026, we spent months performing an autopsy of these failures. Not to fuel controversy, but to understand one thing: why such a linear idea — "give it the documents and then ask it things" — becomes so difficult when it leaves the demo stage and enters the room of real servers.

This six-chapter investigation recounts what we found: silent bugs that deleted entire catalogs, cloud rerankers with incorrect scores, hallucinations born not from the model but from data contaminated years earlier, outgoing invoices that made every query more expensive than its worth, and "popular" frameworks that promise everything in ten lines of code as long as you don't ask too much of them. And in the last chapter, the different direction that someone, quietly, began to take at the beginning of 2026 — an architecture in which the search strategy ceases to be code and becomes a document that anyone in the company can read and modify.

Each chapter stands on its own. If you want to start with the story of the 342 missing tables, or with that of the lying reranker, you are free to do so. But the story as a whole has a moral that only emerges at the end: the corporate RAG — that technical acronym standing for Retrieval-Augmented Generation, meaning "generate answers based on documents you retrieve first" — is not yet a finished product. It is a frontier. And like all frontiers, it has so far been told mostly by salespeople. It's time to listen to those who have lived within it as well.

Chapter 1 — The myth of the ten lines of code

The slide is identical in every AI conference since 2023: "Your company document assistant in 10 lines of code." Below the title, a block of pastel-colored Python showing an open-source library — typically one of those famous American ones, whose name evokes chains of trees or Tibetan llamas — that loads PDFs, splits them up, pastes them into a vector database, and queries them with a language model. In five minutes, you have a chatbot. In five minutes, the applause. In five minutes, an Italian medium-sized company becomes convinced that the problem is solved and that its IT department can handle it in two weeks.

The problem — the one the slide doesn't tell you — is that the demo is built with three well-formatted PDFs, a question tailored to match the content, a stage without real latencies, and a presenter who has tried it all twenty-seven times before going on stage. In reality, company documents are a geological disaster: scans of scans, tables overlapping text, footnotes sneaking into paragraphs, alphanumeric codes like "RH1M" that the chunker cuts in half believing them to be words, images that contain seventy percent of the useful information but that no one actually extracts, and layouts so creative that they require an archaeologist more than a parser.

RAG — this is the idea under the hood — is a great idea. Take the user's question, search your archive for the most relevant documents, pass them to the language model, and get an answer based on those documents. In theory, it elegantly solves the problem of hallucination: the model no longer has to "know" the answer, it just has to "read" it in the pieces you have provided. In practice, every link in the chain — the chunking, the embedding, the retrieval, the reranking, the final generation — has its ways of breaking, and the breakage rarely manifests as a visible error. It manifests as a slightly wrong answer. Then as a completely wrong answer. Then as a manager wondering why they are paying for a system that knows less than the intern.

The open-source frameworks that have popularized RAG are made to demonstrate, not to produce. They are exquisitely elegant concatenations of abstractions in which each layer hides an unstated assumption: that your PDFs have decent OCR, that your photos have already been described, that your tables follow a convention, that the embedding model actually speaks your language (spoiler: many only speak English well), that your archive has already been cleaned of duplicates. When one of these assumptions fails — and at least one always fails, often three — the system doesn't stop working. Worse: it stops working well, but continues to give answers. Fluent, confident answers, and often disconnected from the truth.

There's a phrase that circulates among those who build these systems for a living, and that you'll never read in tutorials: "RAG is easy to do, and difficult to do well." Between the first demo and the production service there's a gap that cannot be bridged by adding GPUs or changing models. It's bridged by understanding an uncomfortable truth: when you build an enterprise RAG, you're not writing code. You're designing a small, stubborn search engine tailored to your documents, with all the editorial choices that this entails — what is noise, what is signal, what needs to be indexed twice, what should be discarded. Only in ten-line tutorials are these choices made for you, once, by someone who has never seen your documents. And they are almost always the wrong ones for your case.