---
topic: distributed-systems
author: Crashtech Editorial
published: Jul 30, 2026 · read: 4 min
---

Retry Without Double-Charging: Idempotency Keys, Delivery Guarantees, and Deduplication

How payment gateways and distributed APIs prevent double charges: idempotency keys, at-least-once delivery, and atomic locking.

– –

In distributed software, network failures are inevitable. A client issues a POST /v1/charges request to process a $50 transaction. The server processes the card, debits the account, and writes the ledger. But just as the server transmits the HTTP 200 response, a transient router blip drops the packet.

From the client’s perspective, the request timed out. The client does not know whether the transaction succeeded, failed, or stalled midway.

If the client automatically retries the request, does the customer get charged $100?

Solving this problem requires implementing idempotency: the mathematical property that executing an operation multiple times produces the exact same side-effects and outcome as executing it once ($f(f(x)) = f(x)$).

While the concept is straightforward, implementing robust idempotency in high-throughput distributed systems demands careful coordination between atomic locks, payload hashing, and state machines.

The Myth of “Exactly-Once” Delivery

Engineers often ask whether a message queue or network protocol can provide “exactly-once delivery.” In unconstrained distributed networks, exactly-once delivery is physically impossible.

Network protocols operate under the constraints of the Two Generals Problem:

  • If a sender retries unacknowledged messages, you get At-Least-Once Delivery (duplicate processing risk).
  • If a sender drops unacknowledged messages to avoid duplicates, you get At-Most-Once Delivery (message loss risk).

What systems refer to as “exactly-once processing” is actually at-least-once delivery combined with idempotent consumer deduplication. The network delivers duplicates; your application logic renders them harmless.

       [Client Request]
              |
              v
       (Network Retry)  ====>  [Idempotency Filter]  ====>  [Business Logic]
       (Network Retry)  ====>  (Deduplicates & Replays)       (Runs EXACTLY ONCE)

Anatomy of the Idempotency Key Pattern

The industry standard pattern (pioneered by payment systems like Stripe and Adyen) relies on client-generated Idempotency Keys passed in HTTP headers:

POST /v1/orders HTTP/1.1
Host: api.crashtech.internal
Idempotency-Key: 9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d
Content-Type: application/json

{
  "amount_cents": 5000,
  "currency": "usd",
  "customer_id": "cust_123"
}

The server manages this request using a four-phase atomic state machine:

                  [Incoming Request with Idempotency Key]
                                    |
                                    v
                     +-----------------------------+
                     | Read Key from Datastore     |
                     +-----------------------------+
                                    |
            +-----------------------+-----------------------+
            | (Key does not exist)                          | (Key exists)
            v                                               v
   +-----------------------+                       +-----------------------+
   | Atomic Insert:        |                       | Check Status:         |
   | status = 'IN_FLIGHT'  |                       | - 'IN_FLIGHT': Lock!  |
   | hash = SHA256(payload)|                       | - 'COMPLETED': Replay!|
   +-----------------------+                       +-----------------------+
            |                                               |
            v                                               v
   [Process Payment Core]                         [Return Cached Response]
            |
            v
   [Update Key: status = 'COMPLETED', body = HTTP 200]

Phase 1: Atomic Lock Acquisition

When the request arrives, the server queries a durable, fast key-value store (such as Redis or PostgreSQL):

  • If the key does not exist, the server inserts a record with status IN_FLIGHT and an expiration TTL (typically 24 hours). This write must be atomic (e.g., using Redis SET key value NX EX 86400 or SQL INSERT ... ON CONFLICT DO NOTHING).
  • If another thread or node tries to process the identical key concurrently, the atomic insert fails, preventing race conditions.

Phase 2: Payload Hash Validation

What happens if a rogue client uses the same idempotency key for two completely different requests?

  • Request A: Buy 1 Laptop ($1,200) with key key_abc
  • Request B: Buy 10 Monitors ($3,000) with key key_abc

Replaying the cached response of Request A for Request B would be catastrophic.

To prevent this, the server computes a cryptographic hash (SHA-256) of the request body and URL parameters upon first arrival and stores it alongside the key. If a subsequent request arrives with an identical key but a mismatched payload hash, the server immediately rejects the call with an HTTP 422 Unprocessable Entity conflict error.

Phase 3: Handling Concurrent In-Flight Retries

If the client issues a retry before the initial request finishes processing:

  • The server detects the key with status IN_FLIGHT.
  • The server does not execute the business logic again.
  • It can either return HTTP 409 Conflict with a Retry-After header, or subscribe to a Redis Pub/Sub channel and block until the original worker completes, returning the finished response directly.

Phase 4: Atomic Commit and Response Caching

Once the core business logic (charging the card, updating inventory) succeeds inside a database transaction:

  • The server saves the HTTP status code and serialized response body into the idempotency record.
  • The record status transitions to COMPLETED.
  • Future retries simply read the cached response from storage and return it instantly without touching the database or payment gateway.

Database-Level Idempotency Patterns

Relying solely on an external Redis cache for idempotency introduces a subtle failure mode: what if Redis crashes or loses power before the database commits?

For critical financial transactions, idempotency must be anchored directly inside the primary database using relational constraints:

1. Unique Natural Keys

Use database-enforced unique constraints:

CREATE TABLE payments (
    id UUID PRIMARY KEY,
    idempotency_key VARCHAR(64) NOT NULL UNIQUE,
    customer_id UUID NOT NULL,
    amount_cents INTEGER NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

If an accidental retry executes INSERT INTO payments ..., the database engine rejects the duplicate write at the B-tree index level with a unique constraint violation error.

2. State-Based Update Guards (Optimistic Locking)

When updating account balances or order statuses, use conditional SQL statements that guarantee idempotency without explicit lock tables:

-- Idempotent status transition
UPDATE orders 
SET status = 'SHIPPED', updated_at = NOW()
WHERE id = 'ord_992' AND status = 'PROCESSING';

If this query runs ten times, it modifies rows on the first call and does nothing on the subsequent nine.

Checklist for Production Idempotency

  1. Deterministic Hashing: Canonicalize JSON bodies before hashing (normalize whitespace and sort object keys) so identical payloads yield identical hashes.
  2. Set an Expiration TTL: Expire idempotency keys after 24 to 72 hours to prevent unbounded database bloat.
  3. Persist the Full Response: Save the status code, headers, and body so the replayed response is indistinguishable from the original.
  4. Isolate Scope: Prefix idempotency keys with the authenticated tenant ID (tenant_123:key_abc) to prevent malicious collisions across multiple users.
Advertisement

/* Comments */