The Hidden Cost of the Cloud

Chapter 5 — The Hidden Cost of the Cloud

The dominant narrative of the last five years says that the cloud is cost-effective, scalable, and simple. For many workloads, this is perfectly true. For enterprise RAGs, increasingly often, it is not. And the moment you realize it is usually at the end of the first quarter, when the invoice arrives.

Let's take the most underestimated item on the bill: the APIs of vision models. When a company ingests its photographic archive into a modern RAG, each photo must be "described" by a multimodal model that extracts content, objects, context, any superimposed text, mood, composition. A large European model provider — still unnamed — offered, in 2025, an excellent model from the Qwen family with 32 billion parameters, at an interesting price per image. The problem, discovered only in the field and after months of production: under load, the provider truncated the responses. Not always, not predictably, not when the testing team was present: randomly. One photo in ten, sometimes one in five, returned with a JSON cut in half, poorly parsed, with partial or null metadata. Times skyrocketed from ten seconds to two hundred seconds per image, without pattern. The bill for the retry — because retries are paid for, each call, even the one the server truncates — was higher than expected. The database was heterogeneous: some photos rich in metadata, others incomplete. And the team couldn't reproduce the problem in test environments, because in test the load was low and everything worked.

The solution came in three parts: a more compact prompt (shorter answers are less likely to be truncated), intelligent retry logic (if the response takes more than fifty seconds and returns empty, retry immediately), and — when the volume justifies it — the ability to skip the cloud altogether and run the vision model on a local GPU, slower but deterministic. The right architecture, from this experience, is not "cloud always" nor "local always". It's "choose for each call, based on what is needed at that moment".

Then there's the chapter of queries. Each user search, in a classic cloud-native RAG, triggers a cascade of paid API calls. One for embedding the question. One for intent classification (what type of question is it?). One for reranking the documents. One for generating the final answer. Each costs a fraction of a cent. For an internal service with fifty users and ten thousand queries per day — which isn't a small number, but not a huge one for a medium-sized company either — the monthly bill comes to figures that would even make a lenient CFO frown. And growth is linear: double the users, double the bill. There are no economies of scale in the tokens consumed, not for you.

There's also a problem that was brewing in 2025 and became central in 2026: every single query sends pieces of company documents — sometimes confidential, sometimes covered by NDAs, sometimes subject to industry regulations — to the servers of an external provider, in jurisdictions that don't always coincide with yours, with log retention policies that aren't always clear. More than one European company, in the last eighteen months, discovered during an audit — usually prompted by a concerned customer or an ISO verification — that their contracts, price lists, and technical specifications had been processed (and potentially logged for "service improvement" purposes) by infrastructure outside the EU. The surprise, usually, cost more than the savings on local hardware that they had sought to avoid.

The alternative isn't the opposite dogma. "No cloud, local only" is as wrong as "cloud always". The alternative is an architecture that allows you to choose, for each single piece of the pipeline — embedding, classification, reranking, vision, final generation — whether to use a cloud model or a local one, and to change your mind in a day, not a quarter. This requires a design where suppliers are interchangeable, where no piece is tied to a specific company's name, where switching from Regolo to Ollama (or vice versa) is a line of configuration, not a rewrite. And this, until recently, was rare. Popular frameworks, despite the facade of "provider-agnostic", were actually very committed to someone.