---
topic: ai-technology
author: Crashtech Editorial
published: Nov 15, 2026 · read: 3 min
---

How RAG Gives LLMs Access to Enterprise Data: Ingestion, Embeddings and Hybrid Search

A production engineering guide to Retrieval-Augmented Generation, exploring semantic chunking, embedding generation, vector indexes, and reranking.

– –

When an employee asks a base LLM: “What is our company’s parental leave policy for contracted engineers?”, the model cannot answer honestly without hallucinating plausible corporate HR boilerplate. RAG converts the query into a targeted search operation against company documentation, presenting authoritative excerpts to the LLM for synthesis.

The RAG Pipeline: Ingestion vs Query Time

A production RAG architecture operates in two distinct decoupled lifecycles:

[ INGESTION LIFECYCLE (Offline / Asynchronous) ]
Document Sinks (Notion, Drive, Jira, PDFs)
         |
         v (Document Parsers & Cleaners)
[ Semantic Chunker: Preserves Headings & Tables ]
         |
         v (Dense Embedding Model: text-embedding-3-large)
[ High-Dimensional Vectors: 1536-d Float Arrays ]
         |
         v (Vector DB: pgvector, Pinecone, Qdrant)
[ HNSW / IVF Vector Index + Metadata ACL Filters ]

--------------------------------------------------------------

[ QUERY LIFECYCLE (Online / Low Latency) ]
User Query: "contractor parental leave"
         |
         +---> Dense Query Embedding ---> Vector Search (Top 50) ---+
         |                                                           |---> [RRF Merge]
         +---> Sparse Tokenization  ---> BM25 Lexical (Top 50) ----+           |
                                                                               v
                                                                   [Top 30 Candidates]
                                                                               |
                                                                   [Cross-Encoder Reranker]
                                                                               |
                                                                   [Top 5 Grounded Chunks]
                                                                               |
                                                                               v
[ Augmented Prompt: Context + Instructions + Query ] ------------> [Generative LLM]
                                                                               |
                                                                               v
                                                                   [Synthesized Answer]

Phase 1: Ingestion and Semantic Chunking

The quality of retrieval is strictly bounded by chunk quality. Feeding garbage text into high-dimensional vector spaces produces garbage matches.

Chunking Pitfalls and Best Practices

  • Fixed-Token Window (Naive): Slicing documents into arbitrary 500-token chunks cuts sentences and disassociates table rows from column headers.
  • Hierarchical Semantic Chunking: Splits text along markdown headers (#, ##), maintaining parent document metadata and section titles across all child chunks.
  • Chunk Size Trade-Offs: Small chunks (128 tokens) produce precise embedding matches but lack surrounding context. Large chunks (1024 tokens) capture full context but dilute semantic vector specificity. Production systems standardize on 256 to 512 tokens with a 10% overlap sliding window.

Phase 2: Embedding Generation and Vector Indexing

Embedding models map textual meaning into continuous geometric vectors. If two sentences discuss similar concepts, their cosine similarity approaches $1.0$:

Cosine Similarity(u, v) = (u * v) / (|u|||u|||u| |u||v|u|||u|)

Approximate Nearest Neighbor (ANN) Indexing

Comparing a query vector against millions of document vectors via brute-force linear scan ($O(N)$) incurs hundreds of milliseconds of latency. Production databases employ Hierarchical Navigable Small World (HNSW) graphs:

  • Organizes vectors into multi-layer hierarchical proximity networks.
  • Traverses top layers with long-range jumps before descending to fine-grained local neighborhoods.
  • Delivers $O(\log N)$ search latency, querying millions of vectors in sub-10 milliseconds.

Phase 3: Hybrid Search and Cross-Encoder Reranking

Relying exclusively on dense vector similarity creates blind spots when searching for exact product serials, error codes, or legal clauses.

Hybrid Fusion with Reciprocal Rank Fusion (RRF)

Production architectures run dense vector search alongside sparse BM25 lexical search, merging candidates via Reciprocal Rank Fusion:

RRF(d) = sum (1) / (60 + r_m(d))

Where $r_m(d)$ is the document’s rank position in search method $m$.

The Two-Stage Cross-Encoder Pass

Bi-encoders compute vector embeddings for queries and documents independently. While fast, they cannot model dynamic token-to-token interactions between the question and the candidate text.

A Cross-Encoder Reranker accepts the (query, chunk) pair concatenated together through a full multi-head attention transformer. It scores cross-token relevance with extreme precision, filtering out irrelevant chunks before constructing the prompt.

Do

  • Store Access Control List (ACL) permission tokens directly in vector index metadata to enforce document access boundaries during retrieval.
  • Pair dense vector embeddings with sparse BM25 lexical search using Reciprocal Rank Fusion for robust keyword coverage.
  • Use cross-encoder rerankers to narrow down candidate context pools to the top 3-5 authoritative chunks.

Don't

  • Do not dump thousands of raw unindexed document tokens directly into long-context LLMs without filtering.
  • Avoid fixed-character chunking that splits markdown tables or code blocks arbitrarily.
  • Never rely on the LLM to filter out sensitive corporate data that the user’s IAM credentials do not permit.

Retrieval-Augmented Generation bridges the gap between static foundational intelligence and private enterprise truth. By orchestrating semantic chunking, hybrid vector retrieval, and precise reranking, engineering teams eliminate hallucinations and ground AI in verifiable reality.

Advertisement

Frequently asked questions

Why are the embedding model and the generation LLM separate models?

Embedding models are specialized bi-encoders trained specifically to map semantic meaning into high-dimensional geometric vector space ($d=768$ or $1536$). Generative LLMs are autoregressive decoders trained to predict the next token given conversational context.

What is semantic chunking and why does fixed-character chunking fail?

Fixed-character chunking slices text mid-sentence or mid-table, severing critical contextual relationships. Semantic chunking respects natural document structure, including headers, paragraphs, table cells, and code blocks, preserving conceptual cohesion.

How does Reciprocal Rank Fusion (RRF) combine dense and sparse search?

BM25 yields keyword matches (ticket IDs, acronyms), while vector search yields conceptual matches. RRF scores documents based on their reciprocal rank positions across both result sets ($RRF(d) = \sum 1 / (60 + r_i)$), producing a robust hybrid candidate ranking.

What is the role of a cross-encoder reranker in reducing LLM hallucination?

Bi-encoder vector search retrieves broad candidate chunks in sub-20ms. A cross-encoder reranker passes the query and candidate chunk together through a full transformer attention layer, filtering out misleading false positives before prompt assembly.

Sources & further reading

/* Comments */