Description
Count UTF-8 bytes preceding a supplied morpheme boundary.
Language-processing records distinguish tokens, code points, UTF-16 positions and UTF-8 byte offsets. Tag grammars and local transition checks do not validate a full annotation sequence. Log-loss units determine perplexity conversion, while identifier suffixes may be textual labels rather than ordinary decimal numbers.
Count UTF-8 bytes preceding a supplied morpheme boundary.
When to use this tool
Use Morpheme Boundary Byte Offset when you already have morphemes, boundaryIndex and need to count UTF-8 bytes preceding a supplied morpheme boundary.
How to use the tool and how it works
- Enter morphemes, boundaryIndex. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read utf8ByteOffset, precedingMorphemeCount. Use the output convention below.
Reject unpaired surrogates in every morpheme; require a safe boundary index 0..morphemes.length. Concatenate preceding morphemes without separators, UTF-8 encode the prefix and return byte length and preceding count.1
UTF-8 encodes a Unicode scalar value in one to four bytes; supplementary characters use surrogate pairs in UTF-16. A byte boundary therefore differs from a JavaScript string-length boundary.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| morphemes | All entries must have paired Unicode surrogates, including entries after the requested boundary. | Required. Array of text; maximum UTF-16 length 8192; maximum items 256. |
| boundaryIndex | Boundary Index | Required. Safe integer; minimum 0. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| utf8ByteOffset | Byte count of concatenated morphemes strictly before boundaryIndex, without separators. | Required. Safe integer. |
| precedingMorphemeCount | Preceding Morpheme Count | Required. Safe integer. |
{
"input": {
"morphemes": [
"un",
"café",
"s"
],
"boundaryIndex": 2
},
"output": {
"utf8ByteOffset": 7,
"precedingMorphemeCount": 2
}
}Limitations and assumptions
- This is a byte offset, not code-point, grapheme or UTF-16 offset. No segmentation, normalization, original-byte spelling, BOM or separator length is inferred.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Validate the complete corpus format, tokenizer, annotation sequence and scoring convention. Use a full parser or model evaluation when needed, preserving the exact offset unit, vocabulary and log base rather than inferring them from a local transformation.
References
Similar or alternative tools
- Corpus Character Offset Newline Adjustment
Adjust a supplied source offset for preceding one-unit newline expansions.
- Entity Span Inclusive Exclusive Crosswalk
Crosswalk supplied inclusive entity-span endpoints to an exclusive end.