Description
Budget UTF-8 bytes in detected seven-bit ESC/CSI/OSC control tokens with lexical completeness evidence; not a complete supported-terminal command or display-width validator.
Terminal escape sequences can encode cursor commands, text styles and other controls inside ordinary text. SGR changes selected graphic attributes; CSI and OSC tokens have different framing. These tools operate on explicit strings and records, with a deliberately limited parser or style model rather than an emulated screen.1
Budget UTF-8 bytes in detected seven-bit ESC/CSI/OSC control tokens with lexical completeness evidence; not a complete supported-terminal command or display-width validator.
When to use this tool
Use this tool to budget UTF-8 bytes in detected seven-bit ESC/CSI/OSC control tokens with lexical completeness evidence; not a complete supported-terminal command or display-width validator.
How to use the tool and how it works
- Enter text using the field definitions below.
- Run the operation. Correct input errors before interpreting an output.
- Read result.tokens, result.controlBytes. Use the ordering, boundary rules and absence conditions described below.
Tokenize ESC-led CSI, OSC and two-character ESC sequences. Return each token’s source span, completeness and UTF-8 byte length, plus their total byte length.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| text | Source text or visible text, including actual escape characters where the operation inspects them. | Text, at most 8,192 UTF-16 code units; the operation’s grammar also applies. |
| Output field | Interpretation |
|---|---|
| result | Object containing the named fields below. |
| result.tokens | Recognized escape-token records with source spans and completeness; byte-budget records also include UTF-8 bytes. |
| result.controlBytes | Sum of UTF-8 byte lengths of the recognized escape tokens. |
{
"input": {
"text": "a\u001b[31mb"
},
"output": {
"result": {
"tokens": [
{
"start": 1,
"end": 6,
"type": "CSI",
"complete": true,
"bytes": 5
}
],
"controlBytes": 5
}
}
}Limitations and assumptions
- This counts the tokenizer’s recognized escape tokens, not every control byte or terminal display cell.
- Source spans use zero-based UTF-16 offsets with exclusive ends; byte totals, where returned, use UTF-8. Escape tokens are represented as JSON escapes in the example and are not executed by this description.
- Every declared top-level input field is required and unexpected fields are rejected. Ordinary strings are limited to 8,192 UTF-16 code units and lists to 128 entries, with tighter operation-specific bounds where stated. JSON text has bounded nesting and node counts.
- Numeric outputs must be finite and no larger in absolute magnitude than the safe-integer limit; operations returning exact large integers use decimal text. Positions are zero-based and spans end exclusively unless the operation explicitly states a different convention.
Alternative or Complementary methods
Use token inspection for source offsets and style replay for the supported attributes. A real terminal emulator is needed to determine screen contents, terminal-specific rendering or device behavior.
References
-
ECMA-48: Control Functions for Coded Character Sets — Ecma International
-
Tool behavior and example: Terminal Supported Control Token Budget — tool.io
Similar or alternative tools
- Terminal CSI Erase Mode Token Parser
Parse complete CSI erase-display/erase-line tokens with modes 0–2 or display-only mode 3; returns lexical evidence without terminal effects.
- Terminal Escape Token Incomplete Tail Finder
Find the first incomplete ESC/CSI/OSC lexical token tail under the documented seven-bit scanner; no terminal emulation or UTF-8 byte-stream decoding.