---
topic: ai-engineering
author: Crashtech Editorial
published: Sep 7, 2026 · read: 4 min
---

Error Handling and Resiliency for LLM Applications: Retries, Fallbacks, and Circuit Breakers

Why HTTP 200 does not equal success in AI systems: classifying transient vs semantic failures, jittered backoff, and circuit breakers.

– –

In traditional web development, error handling is clean and well-understood. If a database query fails, it throws a SQL exception; if an external microservice is down, it returns an HTTP 503 Service Unavailable; if a parameter is missing, the API rejects it with an HTTP 400 Bad Request.

In applications powered by Large Language Models, this paradigm breaks down completely.

An LLM API call can return an HTTP 200 OK status code with low latency, yet represent a catastrophic failure:

  • The model returned valid JSON that completely violates your Pydantic schema.
  • The model hallucinated a fictional product SKU that does not exist in your warehouse.
  • The model got trapped in a repetitive token loop until it hit the token generation ceiling.
  • The model refused a benign customer request due to an over-sensitive safety classifier false positive.

Building production AI applications requires treating the language model as an inherently unreliable third-party component. Resiliency is not about hoping the model does not fail; it is about engineering a multi-tiered safety harness that classifies errors, manages backoff, trips circuit breakers, and degrades gracefully.

+--------------------------------------------------------------------------+
|                  THE THREE CLASSES OF AI FAILURES                        |
|                                                                          |
|  1. Transient Network Failures  : Timeouts, 429 Rate Limits, 503 Outages |
|  2. Permanent Request Failures  : Auth errors, Context length overflow   |
|  3. Semantic Output Failures    : Schema violation, Hallucination, Loops |
+--------------------------------------------------------------------------+

Failure Classification: Knowing How to Respond

The first architectural mistake is treating all errors with the same retry loop. Blindly retrying every failed LLM request burns money, exacerbates provider outages, and loops indefinitely on bad inputs.

Every failure must be categorized into one of three distinct failure buckets:

                  [Incoming Request Exception]
                               |
            +------------------+------------------+
            |                                     |
    (HTTP Status Code)                    (Payload Evaluation)
            |                                     |
   +--------+--------+                           +--------+--------+
   |                 |                           |                 |
(Transient)     (Permanent)                 (Semantic Bad)   (Safety Block)
429 / 503 / Drop  401 / 400 / 404           Invalid JSON     Refusal Error
   |                 |                           |                 |
   v                 v                           v                 v
[Retry w/ Jitter] [Fail Fast to User]      [Repair Parser]  [Fallback Model]

1. Transient Failures (Retry with Exponential Backoff and Jitter)

  • Causes: HTTP 429 Rate Limit (Too Many Requests), HTTP 500/503 Provider Overloaded, socket read timeouts, TCP resets.
  • Action: Retry automatically using Exponential Backoff with Full Jitter:
t_sleep = random(0, min(t_max, t_base * (2 ^ attempt)))
  • Adding randomized jitter prevents the Thundering Herd Problem, where thousands of client workers retry at the exact same second, DDOSing the recovery of the upstream provider.

2. Permanent Failures (Fail Fast)

  • Causes: HTTP 401 Invalid API Key, HTTP 400 Context Length Exceeded (Prompt too large for window), malformed endpoint URLs.
  • Action: Do not retry. A request that exceeds the 128K context window will exceed it again on every subsequent attempt. Fail fast, log the error, and notify the calling service immediately.

3. Semantic Failures (The Model Broke the Contract)

  • Causes: The HTTP call succeeded, but the output text contains unparseable markdown formatting, missing required JSON keys, or hallucinations.
  • Action: Pass the raw text through a local deterministic JSON repair parser (like json-repair). If repair fails, execute a targeted correction retry where the model’s bad output and the parser validation error are appended to context: “Your previous output was missing the required ‘order_id’ key. Fix the JSON and output only valid JSON.”

The Circuit Breaker Pattern for AI Providers

Frontier model providers suffer outages. When an upstream provider experiences a degraded state, your API latency explodes from 800 ms to 30 seconds as requests queue and timeout.

If 10,000 concurrent users hit your application during a provider slowdown, all 10,000 backend threads hang, exhausting database connection pools and bringing down your entire web infrastructure.

You must install a Distributed Circuit Breaker (using Redis or Envoy):

       [Normal State: CLOSED]
                 |
                 | (5 consecutive timeouts or 50% failure rate)
                 v
       [Tripped State: OPEN]
                 |
                 +---> Rejects requests instantly (0ms latency!)
                 +---> Routes all traffic to FALLBACK SECONDARY PROVIDER
                 |
                 | (After 60-second cooldown timer)
                 v
       [Trial State: HALF-OPEN]
                 |
                 +---> Lets 5% of traffic test Primary Provider
                 +---> (If successful: Reset to CLOSED; If fails: Back to OPEN)
  1. Closed: Normal operations. Traffic flows to the Primary Provider (e.g., Claude 3.5 Sonnet).
  2. Open: If failure rate exceeds 20% over a 1-minute window, the breaker trips to OPEN. Subsequent requests do not touch the primary provider; they route instantly to a Fallback Provider (e.g., GPT-4o or DeepSeek-V3).
  3. Half-Open: After a 60-second cooldown, a small probe sample of traffic tests the primary provider. If the probe succeeds, the breaker resets to Closed.

Fallback Design: Graceful Degradation

A fallback must never be an afterthought. A fallback strategy must preserve the core user requirement even when operating under degraded intelligence:

Primary FailureFallback Tier 1Fallback Tier 2
Primary Frontier Model (Claude)Secondary Frontier Model (GPT-4o)Fast Local Model (Llama-3-70B via vLLM)
Complex Multi-Step Agent PlanSimple Single-Turn Direct SummaryCached Static Response / Heuristic Template
Real-Time Vector Search (RAG)Classical BM25 Keyword SearchStandard FAQ Knowledge Lookup

If an automated travel agent fails to generate a custom 5-day personalized itinerary due to API timeouts, the application should not display an error modal. It should display a curated popular itinerary template for that city with a polite banner: “Customized personalization is temporarily running in offline mode.”

Idempotency for Tool-Calling Retries

The most dangerous failure occurs during multi-step tool execution.

An agent decides to refund a customer. It issues a tool call to the billing microservice. The billing service charges the card, but the network drops before the confirmation reaches the agent.

If the agent simply retries the tool loop, it will charge the card a second time.

All agent tool invocations must carry client-generated Idempotency Keys. If an agent retries a tool call due to a transient failure, the payment gateway detects the duplicate key and replays the original success response without modifying accounts.

Resiliency in AI systems is not achieved by finding a smarter model; it is achieved by building a bulletproof deterministic cage around the models you already have.

Advertisement

/* Comments */