Description
Return UTF-8 bytes per Unicode code point for one token.
Language-processing records depend on tokenization, indexing, null sentinels and text representation. A UTF-8 byte offset, UTF-16 index, dependency link or literal feature label can be handled precisely without inferring linguistic correctness or semantic equivalence.
Return UTF-8 bytes per Unicode code point for one token.
When to use this tool
Use Token Byte Character Ratio when checking a supplied linguistic annotation, span, alignment or text-offset convention.
How to use the tool and how it works
- Enter token. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read utf8Bytes, codePoints, ratio, defined. Inspect defined before interpreting the associated values.
Return UTF-8 bytes per Unicode code point for one token. Count encoded bytes and code-point iteration length; ratio divides them, or zero with defined false for empty text.1
UTF-8 bytes, UTF-16 code units and Unicode code points are distinct; grapheme boundaries need separate treatment.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| token | Token | Required. Text; maximum UTF-16 length 8192. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| utf8Bytes | Utf8 Bytes | Required. Safe integer. |
| codePoints | Code Points | Required. Safe integer. |
| ratio | Encoded UTF-8 byte count/code-point count, or zero when defined=false. | Required. Finite number. |
| defined | Whether the stated denominator or information basis permits interpretation of the associated value; inspect before using a placeholder. | Required. Boolean. |
{
"input": {
"token": "Aé🚀"
},
"output": {
"utf8Bytes": 7,
"codePoints": 3,
"ratio": 2.3333333333333335,
"defined": true
}
}Limitations and assumptions
- This is a representation ratio, not tokenizer efficiency or grapheme width; encoding behavior for isolated surrogates follows the implementation.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Alternative or Complementary methods
Use the appropriate tokenizer, Unicode and annotation-format conventions and validate the complete text or tree for full linguistic processing.
References
-
Calculation and input contract: computational-linguistics/token-byte-character-ratio — tool.io
-
Unicode 17.0: conformance and encoding forms — Unicode Consortium
Similar or alternative tools
- Subword Token Byte Offset
Return a subword token's byte offsets in concatenated tokens.
- Token Offset Domain Gap
Slice a supplied string under a code-point or UTF-16 coordinate convention.