Building a RAG App: The Five Decisions That Decide Quality
Model, orchestration, vector storage, document extraction and embeddings. Where RAG apps actually fail, and the choices that matter more than the framework.
On this page
Retrieval-Augmented Generation looks easy in a tutorial: embed some documents, search, paste results into a prompt. Then a user asks a question with a product code in it and the answer is confidently wrong. The model was fine. The retrieval was not.
The five components
- Language model
Understands the question and writes the answer from the retrieved context. Options range from hosted frontier models to open-weight models you run yourself. Picking one is the least permanent decision here.
- Orchestration
Code (or a framework) that handles prompts, retrieval calls, memory, routing and model switching. Frameworks speed up prototypes, but for a simple pipeline a few hundred lines of your own code is often clearer and easier to debug.
- Storage and search
Where embeddings live and how similarity search runs. Options include a relational database with a vector extension such as pgvector, or a dedicated vector database. Keep your metadata (permissions, source, dates) alongside so you can filter.
- Document extraction
Turning PDFs, web pages, slides and tables into clean text. This is the most underrated step. Garbage in extraction (broken tables, merged columns, lost headings) becomes garbage in retrieval.
- Embeddings
Models that turn text into vectors so semantic similarity becomes a distance. The embedding model sets the ceiling on how well your search understands your domain.
Where RAG apps actually fail
When an answer is wrong, print the retrieved chunks. If the right passage is absent, no prompt or model change will fix it. If it is present but ignored, then the prompt or model is the problem. This one check halves the debugging time.
A pragmatic starting design
- One database for documents, metadata and vectors, to keep operations simple until scale proves otherwise.
- Clean extraction with source, section and date stored on every chunk, so answers can cite and filter.
- Moderate overlapping chunks along natural boundaries such as headings, tested at a few sizes.
- Hybrid retrieval followed by an optional reranking step on the top candidates.
- Citations in the answer pointing to retrieved sources, so users can verify.
- A small evaluation set of real questions with known good passages, run on every change.
Do
Invest first in extraction quality, chunking and an evaluation set. Treat the framework and even the model as swappable parts.
Don't
Expect the model to compensate for missing context. If the passage was never retrieved, the best case is a refusal, the worst case is a fluent invention.
Takeaway
RAG quality is mostly retrieval quality, and retrieval quality is decided upstream: how text was extracted, chunked, embedded and searched. Build a small evaluation set, measure whether the right passage comes back, and tune from there. The model is the last thing to blame.
Frequently asked questions
What is the most common reason a RAG app gives bad answers?
Poor retrieval, not a weak model. If the right passage is not in the retrieved context, the model cannot use it, and it may fill the gap with a plausible guess. Check retrieval quality first: are the correct chunks being returned for real questions?
Do I need a dedicated vector database?
Not necessarily. For modest corpora, a relational database with a vector extension such as pgvector can store embeddings next to your other data and support similarity search, simplifying operations. Dedicated vector databases make sense at larger scale or when you need specialised features.
How big should chunks be?
Big enough to carry a complete idea, small enough to stay focused. There is no universal number. Chunks that are too small lose context, chunks that are too large dilute the match and waste the model's context window. Test a few sizes against real questions and measure retrieval.
Why use hybrid search instead of only vector search?
Embeddings capture meaning but can miss exact terms such as product codes, names or error messages, where keyword search excels. Combining both and merging the rankings usually retrieves better results than either alone, especially on technical content.
Sources & further reading
Keep reading
Agent Skills Can Travel Between Tools. Your Permissions Still Cannot
GitHub’s guide to portable agent skills points to reusable workflows, but teams must separately review tools, context, permissions and risk.
Intel and RadixArk’s SGLang Deal Puts the Inference Stack in the Spotlight
Intel and RadixArk say they will optimize SGLang for Intel systems. What the announcement promises—and what operators still need to measure.
Gemini Robotics 2.5 Targets the Awkward Work That Slows Factory Lines
Google DeepMind says Gemini Robotics 2.5 reached 96% success in a factory kitting pilot. What the result does and does not show.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.