Your Image Pipeline Doesn't Have a Storage Problem, It Has an Identity Problem
Uber Eats cut redundant image processing by keying on content hashes instead of URLs — the same fix applies anywhere 'new URL' wrongly means 'new asset.'
On this page
A single stock photo of a soda can, uploaded independently by a few thousand different merchant storefronts, is — to a system that identifies images by URL — a few thousand separate images. Each upload gets its own download, its own processing pass, its own storage slot, its own CDN footprint, purely because each one arrived at a different address. The bytes are identical every time; the system has no way to notice, because it was never asked to compare bytes — only URLs.
The old assumption, and exactly where it breaks
The original design rested on one implicit rule: a new URL means a new image; the same URL means nothing changed. That rule fails in two specific, opposite directions:
- Different URLs, identical content: nothing in the system compares byte content across different upload paths, so the same image reuploaded under a new URL — a different merchant, a different listing, a different context — is treated as entirely new work, every time.
- Same URL, different content: if a merchant updates the image behind an existing URL without changing the URL itself, the old system has no signal to notice — no revalidation, no reprocessing, no cache invalidation. The update is silently dropped.
Both failures trace back to the same root cause: the system was keyed on where an image came from, when the actual question that mattered was what the image actually is.
The fix: identity by content, not by address
- Hash the bytes, not the URL
Every image is identified by a cryptographic hash of its actual content. Two uploads with identical bytes produce the identical hash, completely independent of URL, merchant, or upload context — the system can now recognize genuine duplicates it was previously blind to.
- Key cached results on hash plus processing spec
A processing spec (target format, dimensions, resolution constraints) combined with the content hash forms a single lookup key. If that exact combination has been processed before, the cached result returns immediately — no download, no transformation, no storage write.
- Treat failures as cacheable results too
If an image is too small for a requested resolution, that failure is deterministic — the same hash against the same spec will fail identically every time. Caching the failure under that key means every subsequent identical request skips straight to the known failure, instead of repeating a doomed download and transformation.
- Use Last-Modified to catch changes behind stable URLs
Content hashing alone won’t tell the system to recheck a URL it already has cached. Comparing the server’s current Last-Modified header against the stored value is the cheap signal that decides whether to re-download and re-hash, or trust the cached content — catching real updates without redundant downloads on every request.
The redesigned system keeps identity resolution (which content hash does this URL currently point to), content storage (raw bytes by hash), and processed-output caching (a specific hash-plus-spec combination’s result) as three separate lookups rather than one conflated structure — each map answers exactly one question, which is what makes the whole system reason-about-able and independently cacheable at each stage.
Why this pattern generalizes past image pipelines
The specific mechanism — content hash instead of source address as the identity key — isn’t specific to images at all. Anywhere a system’s “have I seen this before” check is keyed on where something arrived from rather than what it actually is, the same class of redundant-work bug is possible:
Do
Ask, for any caching or deduplication layer, whether the thing you’re keying on (a URL, a filename, an upload timestamp) is actually a proxy for content identity — and if two different keys can legitimately carry identical content, hash the content directly instead.
Don't
Assume a bigger cache or a longer TTL fixes redundant processing caused by an identity mismatch. A cache keyed on the wrong thing just caches the redundant work faster; it doesn’t stop the redundancy from existing in the first place.
The measurable outcome of getting identity right
Reported results from this specific migration: median response latency around 100 milliseconds, P90 under 500 milliseconds, and the large majority of requests resolving entirely from cache without any reprocessing at all — on one of the platform’s highest-volume data paths, delivered as an additive change rather than a full rewrite, in under two months.
The speed of that rollout is itself a signal about the nature of the fix: this wasn’t a new storage system or a new CDN — it was a new identity scheme layered onto existing infrastructure, which is a smaller, safer, faster change than it might sound like from the scale of the savings it produced.
Takeaway
“We have a storage cost problem” and “we have an identity problem” can look identical from the outside — both show up as too much redundant work happening. But the fix for the second one isn’t more capacity, it’s a different question: are you asking “have I seen this address before,” when the question that actually matters is “have I seen this content before.” Content-addressable identity — hash the bytes, cache both results and known failures under that hash, use a cheap signal like a Last-Modified header to catch real changes — is the general answer, and it applies well past image pipelines to anywhere the same content can legitimately arrive by more than one path.
Frequently asked questions
Why does keying an image pipeline by URL cause redundant work?
A URL is an address, not an identity — two different URLs can point to byte-identical images (the same product photo uploaded by different merchants), and a URL-keyed system has no way to notice that overlap, so it downloads, processes, and stores the same bytes repeatedly under each new URL. The system was answering 'have I seen this URL' when the question that actually mattered was 'have I seen these bytes.'
What is content-addressable caching, in plain terms?
Instead of identifying an asset by where it came from (a URL, a filename), you identify it by a cryptographic hash of its actual content. Two uploads with identical bytes produce the identical hash regardless of source, URL, or merchant — so the system can recognize 'I've already processed this exact image' even when it arrives through a completely different path than before.
Why treat processing failures as cacheable results instead of just retrying?
If an image is too small for a requested resolution, that failure is deterministic — retrying the same image against the same processing spec will fail identically every time. Caching the failure under the same hash-plus-spec key means the system recognizes 'this exact combination is guaranteed to fail' and skips the wasted download and processing work on every subsequent identical request, instead of repeating a doomed operation.
How do you detect an image update when the merchant keeps the same URL?
Content hashing alone can't tell you to re-check a URL you've already cached — you need a cheap signal that something might have changed. The HTTP Last-Modified header serves that role: compare the currently stored value against what the server reports now, and only re-download and re-hash when the timestamp has actually changed, avoiding both stale content and unnecessary redundant downloads.
What made this specific migration lower-risk than a typical infrastructure rewrite?
The new pipeline extended the existing system's behavior — content that had already been processed continued to resolve correctly, and the change was additive (a new identification and caching layer) rather than a rip-and-replace of the storage or serving infrastructure. Reported results included a majority of requests resolving from cache without reprocessing, and the whole rollout completed in under two months on one of the platform's highest-volume data paths.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.