Web Crawl Frequency by Site Change Rate Recommender
Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use.
Description
Apply a local planning policy to recommend a web-archiving crawl cadence from the weekly page-change fraction: monthly below 0.5%, weekly below 2%, twice weekly below 8%, and daily otherwise. Adjust the thresholds to match collection risk and crawl budget before operational use.