We were told RAG would be the “adult in the room” for AI – setting validations and guardrails to ensure that only correct information is presented to the user. It was touted as the magic potion that will finally stop LLMs from making things up.
The logic was simple: if you give the model a reliable data source, it’ll stop hallucinating.
Think of RAG as giving AI an open-book exam instead of a memory test.
But anyone actually building with it knows well that the truth is much messier. We have unknowingly traded our problem of hallucination with a whole new set of technical headache.
We used to have a problem of “creative lying” where model hallucinated on large data sets and provided incorrect results but now after RAG, we have another set of problem which we can call as “contextual amnesia” where even though model has access to right data source and has the answer right in front of it, but somehow still manages to ignore it.
RAG is less about making AI smarter, and more about making it honest.
If you thought just plugging in a vector database is the one stop solution for all your problems; it’s time for a reality check.
Below is the 3-point diagnosis: where we will try to understand the problems and their fixes.
Similarity ≠ Answers (and we learned this the hard way)
The Reality: We all fell for the marketing gimmick that vector search is magic. It’s not. It’s just math trying to find answers that sound similar to your question. A real world example would be: you asking it for “Q3 Revenue” but in response it gives you a neatly written paragraph on “Revenue Strategy” with no reference to the actual numbers.
The Fix: Don’t label our legacy methods as “Slop” by default. Use Hybrid Search, and use basic keyword matching with fancy vector search. If you have proper guardrails in place, you will be more confident of the response.
More Context Didn’t Help — It Actually Made Things Worse
The Reality: Most LLMs have “Lost in the middle” problem. If you push 20 documents to a prompt, it doesn’t mean the model will actually read them. Most LLMs have this specific problem – they remember the beginning and the end but they tune out the middle where the actual answer usually lives.
This is the very reason your RAG feels like hit-or-miss.
The Fix: You need to stop dumping top 20 results in the prompt. Use a middleware – a smaller, cheaper model to score those results first and providing top 2-3 results to the LLM that it actually needs.
We need to focus on quality over quantity, every single time.
Messy Data In, Confident Garbage Out
The Reality: We spend weeks obsessing over prompts, but the real villain is usually a messy PDF from 2020. To keep things “manageable,” most people set up their RAG systems to blindly chop documents every 500 characters.
That’s where things go wrong.
This approach often cuts sentences in half, separates tables from their explanations, and strips away the very context the AI needs to understand anything. If the data going into your system is fragmented and nonsensical, no amount of “prompt engineering” is going to save the output.
The Fix:
Pause and look at how you’re actually breaking up your files. Stop using random character counts. Instead, group the chunks by meaning – paragraphs, sections, or clear topics.
One complete thought is far more useful than ten broken pieces of a puzzle. Because if you feed the system garbage, it won’t just give you garbage back—it’ll give you confident garbage, based on a half-sentence it never fully understood.
Final Thought
In my view: “RAG works—but only with the right prescription.”
In simple terms, we can break down the prescription in three medicines:
- Right data – Start with data you’d trust yourself
RAG can only work with what you give it. If the data isn’t clear, up to date and well-structured, no amount of volume is going to help.
In other words: feeding AI fewer, better documents is the right approach rather than pushing everything in.
2. Right retrieval – Be deliberate about what gets retrieved
Pulling in more context doesn’t always help. Pulling in the right context does.
Sometimes one relevant paragraph is better than ten loosely related ones.
3. Right expectations – Stop expecting RAG to be a truth machine
RAG doesn’t magically make AI ‘correct.’ It helps AI stay grounded in your sources.
It reduces guesswork, but it still needs guidance and checks.
None of this is glamorous.
But this is where RAG actually earns its value.
