Token Offset Domain Gap Tool

Description

Slice a supplied string under a code-point or UTF-16 coordinate convention.

Language-processing records depend on tokenization, indexing, null sentinels and text representation. A UTF-8 byte offset, UTF-16 index, dependency link or literal feature label can be handled precisely without inferring linguistic correctness or semantic equivalence.

Slice a supplied string under a code-point or UTF-16 coordinate convention.

When to use this tool

Use Token Offset Domain Gap when checking a supplied linguistic annotation, span, alignment or text-offset convention.

How to use the tool and how it works

Enter sourceText, start, end, codePointOffsets. Use the units and conventions below.
Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
Read slice, length, utf16Start, utf16End. Use the output convention below.

Slice a supplied string under a code-point or UTF-16 coordinate convention. Require 0 <= start <= end <= chosen unit count; return the half-open slice, selected-unit length and equivalent UTF-16 start/end offsets.^[1]

UTF-8 bytes, UTF-16 code units and Unicode code points are distinct; grapheme boundaries need separate treatment.^[2]

Input field	Meaning and units	Accepted input
sourceText	Source Text	Required. Text; maximum UTF-16 length 8192.
start	Start	Required. Safe integer; minimum 0.
end	End	Required. Safe integer.
codePointOffsets	Code Point Offsets	Required. Boolean.

Output field	Interpretation	Returned shape
slice	Slice	Required. Text; maximum UTF-16 length 8192.
length	Length	Required. Safe integer.
utf16Start	Zero-based UTF-16 code-unit start of the chosen-unit slice.	Required. Safe integer.
utf16End	Exclusive UTF-16 code-unit end; may split a surrogate under code-unit mode.	Required. Safe integer.

json

{
  "input": {
    "sourceText": "A😀B",
    "start": 1,
    "end": 2,
    "codePointOffsets": true
  },
  "output": {
    "slice": "😀",
    "length": 1,
    "utf16Start": 1,
    "utf16End": 3
  }
}

Limitations and assumptions

Code-point indexes are not grapheme indexes; UTF-16 slicing can split surrogate pairs and no normalization is applied.
Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.

Alternative or Complementary methods

Use the appropriate tokenizer, Unicode and annotation-format conventions and validate the complete text or tree for full linguistic processing.

References

Calculation and input contract: computational-linguistics/token-offset-domain-gap — tool.io
Unicode 17.0: conformance and encoding forms — Unicode Consortium

Similar or alternative tools

Subword Token Byte Offset
Return a subword token's byte offsets in concatenated tokens.
Token Byte Character Ratio
Return UTF-8 bytes per Unicode code point for one token.