Organization
Three things called “duplicate”
Deduplication turns dangerous the moment three distinct cases get conflated:
- The exact duplicate. Two files with identical bytes, hence the same checksum. The result of a copy, a repeated import, a clumsy restore. Only one deserves keeping.
- The near-duplicate. Two visually very close images: two frames in a row, a JPEG and its RAW, a recropped version, a rescan. The bytes differ, the intent doesn’t.
- The version. Two deliberate states of one image: the original and its edit, the virtual copy, the light print and the contrasty print. These aren’t duplicates, they’re work.
A tool that doesn’t distinguish the three and offers to “remove duplicates” will cost you versions.
How each is detected
Exact duplicates are detected by checksum: two identical SHA-256 values, a certain verdict, no judgement required. It’s the only category that can be handled automatically with confidence.
Near-duplicates are detected by perceptual similarity: a visual signature robust to cropping, resizing, and exposure shifts. The output is a score, therefore a probability — not a verdict. It needs a human pass.
The rule that prevents regret
No automatic deletion on a score. An identical checksum permits an automatic decision; a resemblance never does. Perceptual deduplication must produce a queue of proposals to review, not an action.
Second guardrail: deletion must be reversible for the duration of the review, and collection membership must survive the operation. Removing a duplicate must not drop the image you kept out of three collections.
The RAW+JPEG case
This is the most common false duplicate. The camera produced two files for one exposure. They shouldn’t be deduplicated: they should be paired. The catalog must treat them as two representations of one asset, with a designated reference original.
In practice
- Handle exact duplicates by checksum first: immediate space savings, zero risk.
- Review perceptual matches by hand, in batches, never in bulk.
- Pair RAW+JPEG instead of deduplicating them.
- Verify that deletion preserves the kept image’s memberships.
- Don’t run a deduplication campaign without a verified backup first.
Obscura Flow separates the two mechanisms: exact duplicates by checksum, near-duplicates surfaced for review, and RAW+JPEG handled as a relation between representations rather than as redundancy.