Robots Policy Check
Compare caller-supplied expected and observed records for robots policy check.
Description
A robots-policy checklist tracks recorded crawler-policy requirements.
A robots-policy checklist tracks recorded crawler-policy requirements. Coverage indicates which required labels appear in the supplied observations.1
Compare caller-supplied expected and observed records for robots policy check.
When to use Robots Policy Check
Use this tool to compare caller-supplied expected and observed records for robots policy check.
Prepare the text, recorded observations, aligned figures or editorial criteria requested. Confirm what each label, count, timestamp or supplied flag represents before interpreting the result.
How to use Robots Policy Check and understand its results
- Prepare the text, recorded observations, aligned figures or editorial criteria requested. Confirm what each label, count, timestamp or supplied flag represents before interpreting the result.
- Enter requiredDirectives, observedDirectives using the field table and calculation rules, then run the tool.
- Read coveragePercent, matched, missing, unexpected. The explanation below defines the units, signs, ordering and any flags or placeholder values.
- If the tool reports an input issue, correct the values and run it again. Apply the stated assumptions when interpreting a successfully returned result.
| Input field | Meaning | Value type; see rules below | Requirement |
|---|---|---|---|
| requiredDirectives | Required robots directives | list of string | Required |
| observedDirectives | Observed directives | list of string | Required |
Supply required and observed policy labels. Enter a nonempty required list and an observed list, which may be empty. Trim and lowercase entries for comparison; blank entries are errors. Deduplicate by normalized key. Return matched/missing required labels, unexpected observed labels, and coverage. Required order is preserved; the last trimmed spelling wins for duplicate keys.
coverage%=100×matchedUnique/requiredUnique Units, indexing, boundary conditions and exceptional cases are defined in the calculation rules above.
| Output field | Meaning | Value type |
|---|---|---|
| coveragePercent | Coverage (%) | number; minimum: 0; maximum: 100 |
| matched | Matched items | list of string |
| missing | Missing items | list of string |
| unexpected | Unexpected items | list of string |
{
"input": {
"requiredDirectives": [
"user-agent",
"disallow",
"crawl-delay"
],
"observedDirectives": [
"user-agent",
"disallow"
]
},
"output": {
"coveragePercent": 66.66666666666666,
"matched": [
"user-agent",
"disallow"
],
"missing": [
"crawl-delay"
],
"unexpected": []
}
}Limitations and assumptions
No robots.txt file is fetched or parsed and no crawler behavior, permissions or access restrictions are enforced.
Numeric inputs must be finite. Follow the calculation-specific rules for overflow, underflow and exceptional cases. Lists are limited to 100,000 entries; the rules above may require smaller limits. Missing required fields, extra fields, wrong value types and violations of the calculation rules are input errors.
Alternative or Complementary methods
Use Onboarding Access Checklist: Compare caller-supplied requirements and observations for onboarding access checklist. For a complementary task, use Digital Security Checklist: Compare caller-supplied requirements and observations for digital security checklist.
References
-
Robots Policy Check — tool.io
Similar or alternative tools
- Onboarding Access Checklist
Compare caller-supplied requirements and observations for onboarding access checklist.
- Digital Security Checklist
Compare caller-supplied requirements and observations for digital security checklist.