How Yelp Built Yelp Assistant: Evidence-Grounded RAG, Source Routing, and Hybrid Photo Retrieval
Inside Yelp Assistant: multi-source RAG architecture, sub-3-second responses, evidence-grounded citations, and hybrid photo retrieval over billions of reviews.
On this page
When a hungry diner evaluates a restaurant on Yelp, they rarely want to read through 4,000 individual user reviews. They want immediate answers to specific, nuanced questions:
- “Does this patio have outdoor heaters for chilly evenings?”
- “Is the tiramisu made in-house or brought in frozen?”
- “Can they accommodate severe gluten allergies without cross-contamination?”
Historically, answering these questions required typing keywords into review search bars, clicking through dozens of user photos, and piecing together fragments of subjective feedback.
To automate this, Yelp built Yelp Assistant: an interactive, evidence-grounded conversational AI embedded directly on local business pages. Delivering this feature in production presented a formidable systems engineering challenge: the assistant had to synthesize answers from billions of unstructured reviews, structured business attributes, and millions of user-submitted photos, and return verified, cited answers in less than 3 seconds.
+-------------------------------------------------------------------------------+
| YELP ASSISTANT SYSTEM ARCHITECTURE |
+-------------------------------------------------------------------------------+
[Mobile App: User asks: "Is the patio heated?"]
|
v
[Query Understanding & Intent Classifier (Intent, Entity, Tone)]
|
+---> [Source Routing & Content Fetching API]
| |-- Structured Business Attributes (Hours, Amenities)
| |-- Lexical Search (Elasticsearch BM25 over Reviews)
| |-- Dense Vector Search (HNSW Embeddings over Review Chunks)
| \-- Multimodal CLIP Photo Retrieval (Visual Evidence)
v
[Reranker & Context Assembler (Top-K Chunks + Photo Links)]
|
v
[Evidence-Grounded LLM Generation (Strict Citation Constraints)]
|
v
[Post-Generation Citation Resolver & Hallucination Verifier]
|
v
[Delivered to App: Answer + Verified Review Snippets + Photos (< 3000ms)]
1. The Core Requirement: Evidence Grounding and Zero Hallucination
In local business search, hallucination is unacceptable. If an AI assistant falsely promises a user that a restaurant offers vegan options or has valet parking, the user suffers real-world disruption.
Yelp established an unyielding architectural mandate: every claim made by the assistant must be strictly backed by verifiable citations directly linked to user reviews, business attributes, or photos.
The Citation Invariant
The language model is forbidden from answering using internal parametric knowledge alone. If reviews and attributes do not explicitly confirm whether the patio is heated, the assistant must state:
“I couldn’t find any mentions of patio heaters in recent reviews or amenities for this business.”
2. Multi-Tier Content Ingestion and Freshness Targets
Yelp’s data layer spans diverse data formats updating at completely different cadences:
+-----------------------------------------------------------------------+
| YELP DATA SOURCE FRESHNESS |
+-----------------------------------------------------------------------+
Data Source Update Cadence Storage Engine Index Type
-------------------------------------------------------------------------
Business Attributes Real-Time Cassandra / MySQL Key-Value Lookups
New User Reviews Near Real-Time Kafka -> Elastic BM25 Inverted Index
Review Embeddings Hourly Batches Milvus / Qdrant HNSW Dense Vectors
Photo Tags & Visuals Daily Pipeline S3 + Vector Index CLIP Embeddings
The Content Fetching API (CFAPI)
Rather than having the LLM orchestrator query multiple databases directly, Yelp centralized data access behind a unified Content Fetching API (CFAPI).
When a query arrives, the orchestrator requests business context with a single RPC. CFAPI concurrently fans out requests to:
- The Attribute Service: Retrieves authoritative key-value booleans (
has_outdoor_seating: true,wheelchair_accessible: true). - The Review Inverted Index: Runs lexical BM25 queries matching specific query keywords (
"heaters","patio","blankets"). - The Semantic Vector Index: Runs approximate nearest neighbor (ANN) search over 512-dimension review chunk embeddings, catching semantic equivalents like “they had roaring propane heat lamps outside”.
3. Hybrid Photo Retrieval: Multi-Modal Proof
Text reviews are often ambiguous, but photographs provide definitive proof. Yelp’s users upload hundreds of millions of photos of food, menus, dining rooms, and outdoor spaces.
Yelp Assistant pairs text retrieval with Hybrid Photo Retrieval:
[Query: "Does the patio have heaters?"]
|
v
[Text Embeddings] ----------------------------+
| |
v v
[Search Review Chunks: Hit on Lamp] [CLIP Vision-Text Search over Business Photos]
| |
v v
Review ID: #84920 Photo ID: #491204 (Tagged: "outdoor heat lamp")
| |
+------------------+-------------------+
|
v
[Combined Response: Answer Text + Inline Photo Carousel]
- Yelp passes uploaded business photos through a fine-tuned CLIP (Contrastive Language-Image Pretraining) vision encoder.
- Photo embeddings are indexed per business ID.
- When the assistant detects visual intent, it queries the photo vector index, returning verified images of the outdoor patio alongside the text citation.
4. Latency Engineering: Achieving Sub-3-Second SLAs
Executing query analysis, multi-source retrieval, reranking, and autoregressive LLM generation inside 3,000 milliseconds requires aggressive latency optimization:
+-------------------------------------------------------------------------------+
| YELP ASSISTANT LATENCY BUDGET |
+-------------------------------------------------------------------------------+
Stage Time (ms) Optimization Strategy
---------------------------------------------------------------------------------
1. Intent Classification & Routing 150 ms Lightweight distilled classifier
2. Parallel Content Fetching (CFAPI) 350 ms Concurrent fan-out over gRPC
3. Cross-Encoder Reranking 200 ms ONNX runtime on GPU
4. LLM Generation (First Token) 600 ms KV-caching + Speculative decoding
5. Streaming Autoregressive Output 1200 ms SSE streaming directly to mobile app
6. Citation Validation & Formatting 100 ms Parallel validation thread
---------------------------------------------------------------------------------
Total Time to Complete Response 2600 ms Target: < 3000 ms
Key Latency Levers
- Pre-Filtering by Business ID: Vector search does not search across the global multi-million business dataset; it executes scoped partition filtering on the target
business_id, keeping vector graph traversals under 15ms. - Server-Sent Events (SSE) Streaming: The client does not wait 2.6 seconds for the full paragraph. The mobile app renders the first streamed token within 1,200ms of the user tapping send.
5. Post-Generation Citation Verification
Even with strict prompt boundaries, language models occasionally attribute a genuine fact to the wrong review ID.
To guarantee zero citation drift, Yelp runs a deterministic Citation Resolver prior to finalizing the response:
def verify_citations(generated_answer, retrieved_contexts):
# Parse inline citation markers: e.g., "They offer propane heaters [1]."
citations = extract_markers(generated_answer)
verified_answer = generated_answer
for marker, context_id in citations.items():
source_chunk = retrieved_contexts.get(context_id)
# Run semantic NLI (Natural Language Inference) entailment check
if not check_entailment(premise=source_chunk.text, hypothesis=marker.sentence):
# Strip invalid citation to prevent misleading the user
verified_answer = strip_marker(verified_answer, marker)
log_telemetry("citation_hallucination_detected", context_id)
return verified_answer
If an entailment check detects that review chunk [1] does not actually substantiate the claim, the citation marker is stripped before rendering on the user’s screen.
Production Takeaways
| Architecture Challenge | Naive Approach | Yelp Assistant Engineering Solution |
|---|---|---|
| Hallucination Risk | Prompting model to “be honest” | Hard requirement: citation markers + NLI entailment gate |
| Information Synthesis | Searching reviews only | Multi-source CFAPI: attributes + reviews + CLIP photo search |
| Tail Latency | Sequential database lookups | Concurrent gRPC fan-out + SSE token streaming (< 3s total) |
| Keyword Misses | Pure vector dense search | Hybrid search: BM25 keyword matching + HNSW vector search |
Summary and Key Takeaways
- Grounded RAG requires multi-source ingestion: Real-world entities combine structured facts, unstructured reviews, and visual images; unify them behind a single fetching API.
- Hybrid search is essential for niche facts: Dense embeddings capture semantic equivalents, while BM25 keyword search guarantees exact matches on rare brand names or menu items.
- Bring visual evidence into the conversation: Supplementing text citations with verified CLIP-indexed photographs turns subjective claims into concrete proof.
- Enforce hard programmatic citation verification: Never rely on prompt instructions alone to maintain citation truth; run programmatic entailment checks on generated markers.
- Stream everything under aggressive budgets: Keep business queries under 3,000 milliseconds by scoping vector graphs to business partitions and streaming tokens via Server-Sent Events.


/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.