Duplicate File Hash Set Deduplication Savings MB Calculator
Quantify deduplication savings in megabytes and percent from the total accession size and the size remaining after hash-based duplicate removal.
Description
Calculate hash-deduplication storage savings in megabytes and as a percentage of the original payload.
Quantify deduplication savings in megabytes and percent from the total accession size and the size remaining after hash-based duplicate removal.
When to use Duplicate File Hash Set Deduplication Savings MB Calculator
- Calculate hash-deduplication storage savings in megabytes and as a percentage of the original payload.
- Apply the same explicit rule consistently during ingest, quality control, reporting, or preservation planning.
- Expose records that need human review without silently rewriting the source data.
How the calculation works
The unique-size input is retained payload size, not the count of unique hashes.1
saved = total−unique; savings % = saved/total·100 Interpreting the result
The unique-size input is retained payload size, not the count of unique hashes.
Limitations and assumptions
- Equal hashes are treated as duplicate content; collision policy, metadata overhead, block-level deduplication, compression, and replica requirements are outside the calculation.
- Preserve the original value and record the rule version when using this result in an archival workflow.
References
-
Digital preservation — Wikipedia contributors
Similar or alternative tools
- Born-Digital Disk Image Size TB Aggregator
Aggregate born-digital disk image accession sizes in terabytes into a count, storage total and mean image footprint.
- WARC File Compression Ratio Savings Percent Calculator
Compute the storage compression ratio and percent savings for a WARC web archive from its uncompressed and compressed byte sizes.
- IIIF Image URL Parameter Sequence Formatter (v2)
Assemble a canonical IIIF Image API 2.1 request URL, using the v2 default size keyword 'full' and percent-encoding special characters—including slashes—inside the image identifier.