Robots Allow Checker
Evaluate supplied robots.txt groups for an exact crawler product token and path using RFC 9309 wildcard, longest-rule, and allow-tie semantics.
Description
Robots rules describe crawler access preferences for paths.
Robots rules describe crawler access preferences for paths. The most specific matching rule governs, and equally specific allow and disallow rules favor allow.1
Evaluate supplied robots.txt groups for an exact crawler product token and path using RFC 9309 wildcard, longest-rule, and allow-tie semantics.
When to use Robots Allow Checker
Use this tool to evaluate supplied robots.txt groups for an exact crawler product token and path using RFC 9309 wildcard, longest-rule, and allow-tie semantics.
Choose the URL, decoded source text, header value or explicit byte list requested by this tool. Paste the complete relevant input and follow its parsing and encoding conventions.
How to use Robots Allow Checker and understand its results
- Choose the URL, decoded source text, header value or explicit byte list requested by this tool. Paste the complete relevant input and follow its parsing and encoding conventions.
- Enter robots, productToken, path using the field table and calculation rules, then run the tool.
- Read allowed, matchedRule. The explanation below defines the units, signs, ordering and any flags or placeholder values.
- If the tool reports an input issue, correct the values and run it again. Apply the stated assumptions when interpreting a successfully returned result.
| Input field | Meaning | Value type; see rules below | Requirement |
|---|---|---|---|
| robots | robots.txt contents already acquired by the caller. | string; maximum length: 1000000 | Required |
| productToken | Crawler product token, containing letters, underscores, and hyphens. | string; maximum length: 1000000 | Required |
| path | URL path and optional query, beginning with a slash. | string; maximum length: 1000000 | Required |
Paste acquired robots text, a letters/underscore/hyphen product token and a slash-leading path with optional query. Combine case-insensitive exact-token groups, falling back to * groups. Normalize percent-encoded unreserved bytes and Unicode, then compare case-sensitive path patterns with * and final $. Return allowed plus winning original rule text; no match defaults to allowed.
selected groups=exact product-token matches else wildcard groups; winner=longest matching literal rule, allow wins ties Units, indexing, boundary conditions and exceptional cases are defined in the calculation rules above.
| Output field | Meaning | Value type |
|---|---|---|
| allowed | Whether the selected rules permit the path. | boolean |
| matchedRule | Winning rule text, empty when no rule matches. | string; maximum length: 1000000 |
{
"input": {
"robots": "User-agent: *\nDisallow: /private\nAllow: /private/public\n",
"productToken": "ExampleBot",
"path": "/private/public/article"
},
"output": {
"allowed": true,
"matchedRule": "Allow: /private/public"
}
}Limitations and assumptions
This does not fetch robots.txt or handle HTTP acquisition failures, redirects or caching. Robots preferences do not grant authorization. Unsupported fields and empty/non-slash rules are ignored.
Numeric inputs must be finite. Follow the calculation-specific rules for overflow, underflow and exceptional cases. Extremely large text inputs are rejected. Missing required fields, extra fields, wrong value types and violations of the calculation rules are input errors. A successfully returned false flag is an evaluated outcome; its meaning is explained above.
Alternative or Complementary methods
Use URL Pathname: Extract the pathname of an absolute URL using the WHATWG URL parser, including its normalization rules. For a complementary task, use URL Query Count: Count parsed query parameters in an absolute URL, including repeated names and empty values.
References
-
Robots Allow Checker — tool.io
Similar or alternative tools
- URL Pathname
Extract the pathname of an absolute URL using the WHATWG URL parser, including its normalization rules.
- URL Query Count
Count parsed query parameters in an absolute URL, including repeated names and empty values.