Description
Format supplied source-target token indices as a zero-based alignment pair.
Language-processing records distinguish tokens, code points, UTF-16 positions and UTF-8 byte offsets. Tag grammars and local transition checks do not validate a full annotation sequence. Log-loss units determine perplexity conversion, while identifier suffixes may be textual labels rather than ordinary decimal numbers.
Format supplied source-target token indices as a zero-based alignment pair.
When to use this tool
Use Word Alignment Pair Token when you already have sourceIndex, targetIndex, inputIsOneBased and need to format supplied source-target token indices as a zero-based alignment pair.
How to use the tool and how it works
- Enter sourceIndex, targetIndex, inputIsOneBased. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read zeroBasedToken. Use the output convention below.
Require safe indices at least 1 for one-based input or 0 otherwise; subtract the selected origin from each and join with a hyphen.1
| Input field | Meaning and units | Accepted input |
|---|---|---|
| sourceIndex | Source Index | Required. Safe integer; minimum 0. |
| targetIndex | Target Index | Required. Safe integer; minimum 0. |
| inputIsOneBased | Input Is One Based | Required. Boolean. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| zeroBasedToken | Zero Based Token | Required. Text; maximum UTF-16 length 8192. |
{
"input": {
"sourceIndex": 2,
"targetIndex": 5,
"inputIsOneBased": true
},
"output": {
"zeroBasedToken": "1-4"
}
}Limitations and assumptions
- No sentence length, tokenization, alignment quality or linguistic relation is verified.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Validate the complete corpus format, tokenizer, annotation sequence and scoring convention. Use a full parser or model evaluation when needed, preserving the exact offset unit, vocabulary and log base rather than inferring them from a local transformation.
References
Similar or alternative tools
- Subword Word Boundary Marker Token
Remove one supplied leading subword boundary marker if present.
- Linguistic Tag Attribute Value Token
Format a restricted linguistic attribute-value token.