Subword Token Byte Offset Tool

Description

Return a subword token's byte offsets in concatenated tokens.

Language-processing records depend on tokenization, indexing, null sentinels and text representation. A UTF-8 byte offset, UTF-16 index, dependency link or literal feature label can be handled precisely without inferring linguistic correctness or semantic equivalence.

Return a subword token's byte offsets in concatenated tokens.

When to use this tool

Use Subword Token Byte Offset when checking a supplied linguistic annotation, span, alignment or text-offset convention.

How to use the tool and how it works

Enter tokens, tokenIndex. Use the units and conventions below.
Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
Read startByte, endByte. Use the output convention below.

Return a subword token's byte offsets in concatenated tokens. Require valid tokenIndex and complete Unicode scalar values within each token; sum preceding UTF-8 byte lengths and add selected token length.^[1]

UTF-8 bytes, UTF-16 code units and Unicode code points are distinct; grapheme boundaries need separate treatment.^[2]

Input field	Meaning and units	Accepted input
tokens	Tokens	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.
tokenIndex	Token Index	Required. Safe integer; minimum 0.

Output field	Interpretation	Returned shape
startByte	Zero-based UTF-8 byte offset after concatenating prior tokens without separators.	Required. Safe integer.
endByte	Exclusive UTF-8 byte offset of selected token, on that same concatenation basis.	Required. Safe integer.

json

{
  "input": {
    "tokens": [
      "A",
      "é",
      "🚀"
    ],
    "tokenIndex": 2
  },
  "output": {
    "startByte": 3,
    "endByte": 7
  }
}

Limitations and assumptions

No separators are inserted; tokenization, original-text correspondence and normalization are not verified.
Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.

Alternative or Complementary methods

Use the appropriate tokenizer, Unicode and annotation-format conventions and validate the complete text or tree for full linguistic processing.

References

Calculation and input contract: computational-linguistics/subword-token-byte-offset — tool.io
Unicode 17.0: conformance and encoding forms — Unicode Consortium

Similar or alternative tools

Token Byte Character Ratio
Return UTF-8 bytes per Unicode code point for one token.
Token Offset Domain Gap
Slice a supplied string under a code-point or UTF-16 coordinate convention.