Image Similarity Detection

Perceptual hashing finds visually similar images -- duplicates, near-duplicates, and related images -- even when they differ in resolution, compression, or minor edits.

How It Works

Perceptual Hashing

Unlike cryptographic hashes (SHA1, MD5) which produce completely different outputs for any change, perceptual hashes produce similar outputs for visually similar images.

Three perceptual hashes are computed from a single decode:

Hash Type	Role
Perception Hash (pHash)	Primary matcher. A DCT-based hash whose value is indexed in four chunk columns for fast database-side candidate lookup
Difference Hash (dHash)	Secondary. Compares brightness gradients between adjacent pixels
Average Hash (aHash)	Secondary. Compares the average brightness of image blocks

The pHash drives similarity matching. For each newly hashed image the database finds candidates that share an indexed pHash chunk, then verifies each candidate with a full-width Hamming distance. The dHash and aHash distances are recorded alongside every stored pair; the aHash distance also feeds a secondary guard (see -hash-ahash-threshold) against solid-color false positives.

Before hashing, the image is normalized: EXIF orientation is applied, and any alpha channel is flattened onto white, so a rotated or transparent copy hashes like its upright or white-matted twin.

Hash Status

Every hashed row carries one of three statuses:

Status	Meaning
ok	Hashed successfully and eligible for matching
flat	Near-zero pixel variance (solid color, blank scan)
failed	The file could not be decoded (corrupt, missing, unsupported)

flat and failed rows are excluded from v2 matching entirely: neither is returned as a candidate, and a flat probe is never matched at all. That exclusion, rather than the aHash check, is what keeps solid-color images out of v2 results.

Hamming Distance

Similarity is measured by Hamming distance - the number of bits that differ between two hashes. Lower distance means more similar:

Hamming Distance	Interpretation
0	Identical images (perceptually)
1-5	Near-duplicates (same image, minor edits)
6-10	Similar images (same subject, different versions)
11-15	Loosely related (similar composition)
16+	Different images

Background Hash Worker

A background worker automatically processes images and calculates their hashes.

What Gets Processed

The hash worker processes resources with these content types:

• image/jpeg
• image/png
• image/gif
• image/webp

Other file types are skipped.

Processing Flow

1. Batch discovery - The worker finds images without hashes
2. Hash calculation - Workers compute the pHash, dHash, and aHash for each image
3. Cache update - New hashes are added to the in-memory cache
4. Similarity detection - The new pHash is matched against indexed hashes to find candidates, verified by full-width Hamming distance
5. Persistence - Similar pairs are stored in the database

Worker Configuration

Configure the hash worker using command-line flags or environment variables:

Flag	Env Variable	Default	Description
-hash-worker-count	HASH_WORKER_COUNT	4	Concurrent workers
-hash-batch-size	HASH_BATCH_SIZE	500	Images per batch
-hash-poll-interval	HASH_POLL_INTERVAL	1m	Time between batches
-hash-similarity-threshold	HASH_SIMILARITY_THRESHOLD	10	Max Hamming distance
-hash-ahash-threshold	HASH_AHASH_THRESHOLD	5	Max aHash Hamming distance for the secondary solid-color false-positive check; 0 disables it
-hash-worker-disabled	HASH_WORKER_DISABLED=1	false	Disable entirely
-hash-cache-size	HASH_CACHE_SIZE	100000	Maximum entries in the LRU similarity cache

Tuning Examples

High-performance setup (fast processing, more strict matching):

./mahresources \
  -hash-worker-count=8 \
  -hash-batch-size=1000 \
  -hash-poll-interval=30s \
  -hash-similarity-threshold=8 \
  ...

Resource-constrained setup (slower, gentler on resources):

./mahresources \
  -hash-worker-count=1 \
  -hash-batch-size=100 \
  -hash-poll-interval=5m \
  ...

Disabled (no background processing):

./mahresources -hash-worker-disabled ...

Similarity Threshold Configuration

The -hash-similarity-threshold setting controls how similar images must be to be considered matches:

Threshold	Effect
5	Strict - only near-identical images match
10 (default)	Balanced - finds similar images with variations
11 (maximum)	Loosest, every stored v2 pair matches

v2 perceptual-hash pairs are stored only up to distance 11, so values above that find nothing further.

Choose based on your use case:

• Deduplication - Use a low threshold (5-8) to find true duplicates
• Related images - Use default (10) for variations like crops, resizes
• Broad discovery - Use a higher threshold, up to the maximum of 11, to find related content

Neither threshold gates what is stored. Both are read-time filters over the stored pairs, editable at /admin/settings as hash_similarity_threshold and hash_ahash_threshold, and a change takes effect on the next comparison with no restart and no recompute.

Viewing Similar Images

On any resource's detail page, if similar images exist, you will see a Similar Resources section showing:

• Thumbnails of all similar images
• Links to each similar resource
• A Compare link per similar resource, opening the compare view for that pair (/resource/compare?r1=...&r2=...)
• A form to merge similar resources into one

Finding Images with Similarities

Use the resource search with the filter:

/resources?ShowWithSimilar=true

This shows only resources that have at least one similar image detected.

Merging Duplicates

When you find duplicates, you can merge them:

1. Navigate to the resource you want to keep (the "winner")
2. Find the Similar Resources section
3. Click Merge Others To This
4. Confirm the action

Merging:

• Keeps the winner resource with all its metadata
• Transfers all tags, notes, and group associations from merged resources
• Reassigns and renumbers every merged resource's version records onto the winner, so the winner's history includes them
• When the merge is run with the keep-as-version option, additionally saves each merged resource's current file as a new older version of the winner before deletion
• Deletes the merged resources

warning

Merging is permanent -- the merged resources are deleted. Verify that the winner resource is the one you intend to keep.

Cache Warming

At startup, the hash worker loads existing hashes into the LRU cache in pages of up to 50,000 entries. This pre-populates the cache so similarity detection is effective immediately without waiting for a full batch cycle.

The cache size is controlled by -hash-cache-size (default: 100,000 entries). If your collection exceeds this limit, older entries are evicted and may not participate in similarity comparisons until they cycle back through batch processing.

Failed Hash Handling

If hashing fails for a Resource (corrupt image, unsupported encoding), the worker stores a placeholder hash record marked failed. This prevents the Resource from being retried on every batch cycle. Clearing that marker re-queues the row: see Operator actions.

Operator Actions

Three controls maintain the hash data without a restart.

Recompute similarity pairs (POST /v1/admin/similarity/recompute, or mr admin similarity recompute) deletes every v2 similarity pair and rebuilds it from the stored hashes. It decodes no images, so it is cheap enough to run after an algorithm or threshold change. A second request while one is already running is refused.

Retry failed hashes (POST /v1/admin/similarity/retry-failed, or mr admin similarity retry-failed) clears the failed marker so the backfill attempts those rows again, and reports how many rows it reset.

Pause the backfill by setting the hash_backfill_paused runtime setting to 1. It stops the incremental v2 backfill without disabling the whole hash worker; 0 resumes it.

See mr admin similarity and Runtime Settings.

Memory Considerations

The hash worker maintains an in-memory LRU cache of image hashes for fast similarity lookups. Memory usage depends on your image count and the -hash-cache-size setting.

The figures below count only the raw hash payload per entry. Actual heap use is several times higher once Go map buckets and LRU node bookkeeping are included, so treat these as a lower bound rather than a budget:

Image Count	Raw Hash Data (lower bound)
10,000	~0.2 MB
100,000	~2.4 MB
1,000,000	~24 MB
10,000,000	~240 MB

The cache is loaded at startup and updated incrementally. Cap it with -hash-cache-size when resident memory matters on your deployment.

On-Upload Processing

When you upload a new image, it is queued for immediate hash processing. This means:

1. Upload completes and resource is created
2. Resource ID is added to the hash queue
3. Worker processes it (usually within seconds)
4. Similar images appear on the resource page

If the queue is full (1000 items), new uploads fall back to batch processing on the next poll interval.

Hash Migration

If you have images that were uploaded before hash calculation was available, the hash worker automatically processes them during its batch cycles. No manual intervention is required.

The worker also handles migration of hash format changes transparently. The current storage format uses int64 for efficient Hamming distance calculation. Legacy string-format hashes are still supported and migrated automatically.

A second migration runs alongside it: the v2 backfill re-hashes existing rows that have no hash_version yet, filling in the int64 pHash and the four indexed chunk columns that drive candidate lookup. It processes one batch per cycle, newest resources first, and is fully resumable, because the "no hash_version" predicate is its own cursor. Pause it with the hash_backfill_paused runtime setting (see Operator actions).

Troubleshooting

Similar images not appearing

1. Check that the hash worker is running (not disabled)
2. Wait for the next poll interval
3. Check logs for processing errors
4. Verify the image format is supported

Too many false positives

Lower the similarity threshold:

./mahresources -hash-similarity-threshold=6 ...

Missing obvious duplicates

Raise the similarity threshold:

./mahresources -hash-similarity-threshold=11 ...

High memory usage

If the hash cache is too large:

1. Reduce the cache size: -hash-cache-size=50000
2. Disable the worker if not needed: -hash-worker-disabled
3. Add more system memory