Web Crawl Frequency by Site Change Rate Recommender
Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use.
Description
Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use.
Web Crawl Frequency by Site Change Rate Recommender: Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use.
When to use Web Crawl Frequency by Site Change Rate Recommender
Use the result while planning ingest, digitization, quality control, reporting, or preservation work where the stated rule needs to be repeatable and reviewable.
- Changed pages
- Pages that change in a typical week.
- Total pages
- Total pages tracked in the crawl target.
How Web Crawl Frequency by Site Change Rate Recommender works
Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use. The tool evaluates the supplied inputs together and returns the named outputs below; it does not infer omitted operating conditions or change the units shown.1
- Change fraction
- Share of the collection changing per week.
- Cadence tier
- Recommended crawl cadence tier.
- Crawls per week
- Recommended crawls per week for the tier.
Limitations and assumptions
- Repository policies, format profiles, equipment capabilities, and descriptive standards vary. Preserve the source value and record the rule version instead of treating this utility as an authority-record or preservation-system validator.
- Use finite inputs in the displayed units and preserve more precision than the final presentation requires. Independently verify safety-critical, financial, compliance, or production decisions.
Alternative or Complementary approaches
Complement the calculation with format validation, fixity checks, provenance records, and human review. For policy decisions, apply the profile adopted by the repository rather than an undocumented local convention.
References
-
Web archiving — Wikipedia contributors
Similar or alternative tools
- Migration Cycle Interval Years by Format Class Lookup
Look up an illustrative local review-cycle interval by broad format class. These fixed values are planning defaults, not a preservation standard or a substitute for format-risk monitoring; migration should follow an institution's documented policy and evidence.
- Format Obsolescence Watchlist Risk Score Rater
Rate preservation format obsolescence risk on a capped 0-100 scale (two points per year since release plus twenty for proprietary formats, twenty-five when emulation is required and fifteen when validation tooling is lacking) with low, moderate, high and critical tiers.
- OCR Confidence Score Below Threshold Word Counter
Count OCR word confidence scores that fall below a review threshold and report the affected fraction of the transcribed words.