---
topic: ai-technology
author: Crashtech Editorial
published: Nov 17, 2026 · read: 4 min
---

How Yelp Built Yelp Assistant: Evidence-Grounded RAG, Source Routing, and Hybrid Photo Retrieval

Inside Yelp Assistant: multi-source RAG architecture, sub-3-second responses, evidence-grounded citations, and hybrid photo retrieval over billions of reviews.

– –

When a hungry diner evaluates a restaurant on Yelp, they rarely want to read through 4,000 individual user reviews. They want immediate answers to specific, nuanced questions:

  • “Does this patio have outdoor heaters for chilly evenings?”
  • “Is the tiramisu made in-house or brought in frozen?”
  • “Can they accommodate severe gluten allergies without cross-contamination?”

Historically, answering these questions required typing keywords into review search bars, clicking through dozens of user photos, and piecing together fragments of subjective feedback.

To automate this, Yelp built Yelp Assistant: an interactive, evidence-grounded conversational AI embedded directly on local business pages. Delivering this feature in production presented a formidable systems engineering challenge: the assistant had to synthesize answers from billions of unstructured reviews, structured business attributes, and millions of user-submitted photos, and return verified, cited answers in less than 3 seconds.

+-------------------------------------------------------------------------------+
|                       YELP ASSISTANT SYSTEM ARCHITECTURE                      |
+-------------------------------------------------------------------------------+

[Mobile App: User asks: "Is the patio heated?"]
       |
       v
[Query Understanding & Intent Classifier (Intent, Entity, Tone)]
       |
       +---> [Source Routing & Content Fetching API]
       |        |-- Structured Business Attributes (Hours, Amenities)
       |        |-- Lexical Search (Elasticsearch BM25 over Reviews)
       |        |-- Dense Vector Search (HNSW Embeddings over Review Chunks)
       |        \-- Multimodal CLIP Photo Retrieval (Visual Evidence)
       v
[Reranker & Context Assembler (Top-K Chunks + Photo Links)]
       |
       v
[Evidence-Grounded LLM Generation (Strict Citation Constraints)]
       |
       v
[Post-Generation Citation Resolver & Hallucination Verifier]
       |
       v
[Delivered to App: Answer + Verified Review Snippets + Photos (< 3000ms)]

1. The Core Requirement: Evidence Grounding and Zero Hallucination

In local business search, hallucination is unacceptable. If an AI assistant falsely promises a user that a restaurant offers vegan options or has valet parking, the user suffers real-world disruption.

Yelp established an unyielding architectural mandate: every claim made by the assistant must be strictly backed by verifiable citations directly linked to user reviews, business attributes, or photos.

The Citation Invariant

The language model is forbidden from answering using internal parametric knowledge alone. If reviews and attributes do not explicitly confirm whether the patio is heated, the assistant must state:

“I couldn’t find any mentions of patio heaters in recent reviews or amenities for this business.”


2. Multi-Tier Content Ingestion and Freshness Targets

Yelp’s data layer spans diverse data formats updating at completely different cadences:

+-----------------------------------------------------------------------+
|                      YELP DATA SOURCE FRESHNESS                       |
+-----------------------------------------------------------------------+

Data Source              Update Cadence    Storage Engine        Index Type
-------------------------------------------------------------------------
Business Attributes      Real-Time         Cassandra / MySQL     Key-Value Lookups
New User Reviews         Near Real-Time    Kafka -> Elastic      BM25 Inverted Index
Review Embeddings        Hourly Batches    Milvus / Qdrant       HNSW Dense Vectors
Photo Tags & Visuals     Daily Pipeline    S3 + Vector Index     CLIP Embeddings

The Content Fetching API (CFAPI)

Rather than having the LLM orchestrator query multiple databases directly, Yelp centralized data access behind a unified Content Fetching API (CFAPI).

When a query arrives, the orchestrator requests business context with a single RPC. CFAPI concurrently fans out requests to:

  1. The Attribute Service: Retrieves authoritative key-value booleans (has_outdoor_seating: true, wheelchair_accessible: true).
  2. The Review Inverted Index: Runs lexical BM25 queries matching specific query keywords ("heaters", "patio", "blankets").
  3. The Semantic Vector Index: Runs approximate nearest neighbor (ANN) search over 512-dimension review chunk embeddings, catching semantic equivalents like “they had roaring propane heat lamps outside”.

3. Hybrid Photo Retrieval: Multi-Modal Proof

Text reviews are often ambiguous, but photographs provide definitive proof. Yelp’s users upload hundreds of millions of photos of food, menus, dining rooms, and outdoor spaces.

Yelp Assistant pairs text retrieval with Hybrid Photo Retrieval:

[Query: "Does the patio have heaters?"]
       |
       v
[Text Embeddings] ----------------------------+
       |                                      |
       v                                      v
[Search Review Chunks: Hit on Lamp]       [CLIP Vision-Text Search over Business Photos]
       |                                      |
       v                                      v
Review ID: #84920                         Photo ID: #491204 (Tagged: "outdoor heat lamp")
       |                                      |
       +------------------+-------------------+
                          |
                          v
        [Combined Response: Answer Text + Inline Photo Carousel]
  1. Yelp passes uploaded business photos through a fine-tuned CLIP (Contrastive Language-Image Pretraining) vision encoder.
  2. Photo embeddings are indexed per business ID.
  3. When the assistant detects visual intent, it queries the photo vector index, returning verified images of the outdoor patio alongside the text citation.

4. Latency Engineering: Achieving Sub-3-Second SLAs

Executing query analysis, multi-source retrieval, reranking, and autoregressive LLM generation inside 3,000 milliseconds requires aggressive latency optimization:

+-------------------------------------------------------------------------------+
|                       YELP ASSISTANT LATENCY BUDGET                           |
+-------------------------------------------------------------------------------+

Stage                                  Time (ms)    Optimization Strategy
---------------------------------------------------------------------------------
1. Intent Classification & Routing      150 ms      Lightweight distilled classifier
2. Parallel Content Fetching (CFAPI)    350 ms      Concurrent fan-out over gRPC
3. Cross-Encoder Reranking              200 ms      ONNX runtime on GPU
4. LLM Generation (First Token)         600 ms      KV-caching + Speculative decoding
5. Streaming Autoregressive Output     1200 ms      SSE streaming directly to mobile app
6. Citation Validation & Formatting     100 ms      Parallel validation thread
---------------------------------------------------------------------------------
Total Time to Complete Response        2600 ms      Target: < 3000 ms

Key Latency Levers

  • Pre-Filtering by Business ID: Vector search does not search across the global multi-million business dataset; it executes scoped partition filtering on the target business_id, keeping vector graph traversals under 15ms.
  • Server-Sent Events (SSE) Streaming: The client does not wait 2.6 seconds for the full paragraph. The mobile app renders the first streamed token within 1,200ms of the user tapping send.

5. Post-Generation Citation Verification

Even with strict prompt boundaries, language models occasionally attribute a genuine fact to the wrong review ID.

To guarantee zero citation drift, Yelp runs a deterministic Citation Resolver prior to finalizing the response:

def verify_citations(generated_answer, retrieved_contexts):
    # Parse inline citation markers: e.g., "They offer propane heaters [1]."
    citations = extract_markers(generated_answer)
    
    verified_answer = generated_answer
    for marker, context_id in citations.items():
        source_chunk = retrieved_contexts.get(context_id)
        # Run semantic NLI (Natural Language Inference) entailment check
        if not check_entailment(premise=source_chunk.text, hypothesis=marker.sentence):
            # Strip invalid citation to prevent misleading the user
            verified_answer = strip_marker(verified_answer, marker)
            log_telemetry("citation_hallucination_detected", context_id)
            
    return verified_answer

If an entailment check detects that review chunk [1] does not actually substantiate the claim, the citation marker is stripped before rendering on the user’s screen.


Production Takeaways

Architecture ChallengeNaive ApproachYelp Assistant Engineering Solution
Hallucination RiskPrompting model to “be honest”Hard requirement: citation markers + NLI entailment gate
Information SynthesisSearching reviews onlyMulti-source CFAPI: attributes + reviews + CLIP photo search
Tail LatencySequential database lookupsConcurrent gRPC fan-out + SSE token streaming (< 3s total)
Keyword MissesPure vector dense searchHybrid search: BM25 keyword matching + HNSW vector search

Summary and Key Takeaways

  1. Grounded RAG requires multi-source ingestion: Real-world entities combine structured facts, unstructured reviews, and visual images; unify them behind a single fetching API.
  2. Hybrid search is essential for niche facts: Dense embeddings capture semantic equivalents, while BM25 keyword search guarantees exact matches on rare brand names or menu items.
  3. Bring visual evidence into the conversation: Supplementing text citations with verified CLIP-indexed photographs turns subjective claims into concrete proof.
  4. Enforce hard programmatic citation verification: Never rely on prompt instructions alone to maintain citation truth; run programmatic entailment checks on generated markers.
  5. Stream everything under aggressive budgets: Keep business queries under 3,000 milliseconds by scoping vector graphs to business partitions and streaming tokens via Server-Sent Events.
Advertisement

/* Comments */