Description
Lowercase the name, remove the value from the first # onward, trim its edges, and return directive and value.
Parsing depends on the chosen grammar, character encoding, offset unit, and treatment of escapes and delimiters. UTF-16 code-unit positions, Unicode code points, and UTF-8 byte offsets are not interchangeable. A boundary or token check supplies one part of a parser rather than validating an entire file.
Lowercase the name, remove the value from the first # onward, trim its edges, and return directive and value.
When to use this tool
Use Robots Directive Colon Value Token when you need the stated token boundary, offset, prefix, delimiter, or count component inside a larger parsing workflow.
How to use the tool and how it works
- Enter line. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read directive, value. Use the output convention below.
Accept an ASCII-letter directive name with letters/hyphens followed by optional spaces/tabs, a colon, and a value without CR or LF. Lowercase the name, remove the value from the first # onward, trim its edges, and return directive and value.1
| Input field | Meaning and units | Accepted input |
|---|---|---|
| line | One supplied line under the stated newline policy. | Required. Text; maximum UTF-16 length 8192. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| directive | Directive | Required. Text; maximum UTF-16 length 8192. |
| value | Value | Required. Text; maximum UTF-16 length 8192. |
{
"input": {
"line": "Disallow: /private # comment"
},
"output": {
"directive": "disallow",
"value": "/private"
}
}Limitations and assumptions
- Unknown directive names are accepted and percent-encoded text is not decoded. This does not parse robots.txt groups, select a user agent, interpret matching rules, or establish permission to access a resource.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Alternative or Complementary methods
Choose a parser for the intended complete grammar when context, nesting, encodings, or semantic validation matters. Keep byte, code-unit, and code-point coordinates explicit when combining results.
References
Similar or alternative tools
- Parsing: Ini Key Assignment Delimiter Position
Locate the first explicitly allowed equals/colon delimiter in one restricted unquoted INI key line; no sections, escaped delimiters, comments or continuation syntax.
- Parsing: Dotenv Single Assignment Split
Split a restricted dotenv assignment at its first equals sign, retaining raw value and optional export prefix without evaluating quoting, interpolation or comments.