Reference Glossary
Duplicate detection
The DAM's ability to identify identical or near-identical files on upload, usually via file hashing or perceptual image hashing, to stop redundant copies from piling up.
Why it matters in a DAM
Exact duplicates — the same file uploaded twice — are caught with a simple checksum, but a DAM also needs perceptual hashing to catch a re-saved, recompressed, or resized copy of the same photo, which won't share a file hash at all. Without both checks, the same hero image ends up stored under five different filenames, each tagged separately and each drifting out of sync whenever the original gets updated.
A worked example
Common mistake
Duplicate detection is trusted to catch everything automatically, so nobody spot-checks the results — but perceptual hashing typically misses images that were cropped, flipped, or color-corrected, so visually distinct but still-redundant copies keep slipping through.
Two different mechanisms are usually bundled under ‘duplicate detection.’ A cryptographic file hash (MD5 or SHA) flags exact byte-for-byte duplicates instantly and cheaply, but breaks the moment a single pixel or compression setting changes — a JPEG re-saved at a different quality level produces a completely different hash even though it looks identical. Perceptual hashing solves that by fingerprinting the image’s visual content instead of its bytes: it converts the image to grayscale, reduces it to a small grid, and encodes the low-frequency structure as a compact fingerprint, so two visually similar images produce hashes that are close to each other rather than only matching exactly.
That distinction is what lets a DAM catch the actual duplicate problem teams have — not two identical files, but the same photo re-exported at five different sizes and compression levels by five different people over the years, none of which share a checksum.
Perceptual hashing has real limits: it’s tuned for mild variations like resizing and recompression, not for a flipped, heavily cropped, or color-graded version of the same shot, which can produce a hash different enough to be missed entirely. Most DAMs handle that gap by flagging likely matches for a human to confirm rather than auto-merging, which is the safer default given how easy it is for the algorithm to both over- and under-match.
Frequently asked
What are the two mechanisms usually bundled under 'duplicate detection'?
A cryptographic file hash (MD5/SHA) that flags exact byte-for-byte duplicates instantly, and perceptual hashing that fingerprints an image's visual content so a resized or recompressed copy still matches even though the bytes differ.
Why doesn't a file hash catch a re-saved JPEG of the same photo?
A cryptographic hash breaks the moment a single pixel or compression setting changes — a JPEG re-saved at a different quality level produces a completely different hash even though it looks identical.
How does perceptual hashing work, roughly?
It converts the image to grayscale, reduces it to a small grid, and encodes the low-frequency structure as a compact fingerprint, so visually similar images produce hashes that are close to each other rather than requiring an exact match.
What kinds of duplicates does perceptual hashing typically miss?
It's tuned for mild variations like resizing and recompression, not for a flipped, heavily cropped, or color-graded version of the same shot, which can produce a hash different enough to be missed entirely.
Should duplicate matches be auto-merged or flagged for review?
Most DAMs flag likely matches for a human to confirm rather than auto-merging, which is the safer default given how easily the algorithm can both over- and under-match.
What's the practical duplicate problem duplicate detection is designed to catch?
Not two identical files, but the same photo re-exported at five different sizes and compression levels by five different people over the years, none of which share a checksum.
Sources
- Perceptual hashing (pHash) converts an image into a compact fingerprint based on its visual structure, so a resized or recompressed copy of the same photo can still produce a matching or near-matching hash even though the file bytes differ. checked 2026-08-07 — Cloudinary Engineering Blog