Description
Remove one supplied leading subword boundary marker if present.
Language-processing records distinguish tokens, code points, UTF-16 positions and UTF-8 byte offsets. Tag grammars and local transition checks do not validate a full annotation sequence. Log-loss units determine perplexity conversion, while identifier suffixes may be textual labels rather than ordinary decimal numbers.
Remove one supplied leading subword boundary marker if present.
When to use this tool
Use Subword Word Boundary Marker Token when you already have tokenText, suppliedBoundaryMarker and need to remove one supplied leading subword boundary marker if present.
How to use the tool and how it works
- Enter tokenText, suppliedBoundaryMarker. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read hasMarker, unmarkedText. Inspect hasMarker before interpreting the associated values.
Require marker nonempty; test startsWith and remove exactly its UTF-16 length from the prefix, otherwise return the original token.1
| Input field | Meaning and units | Accepted input |
|---|---|---|
| tokenText | Token Text | Required. Text; maximum UTF-16 length 8192. |
| suppliedBoundaryMarker | Supplied Boundary Marker | Required. Text; minimum UTF-16 length 1; maximum UTF-16 length 8192. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| hasMarker | Has Marker | Required. Boolean. |
| unmarkedText | Remove exactly one leading suppliedBoundaryMarker when present, using JavaScript string slicing. | Required. Text; maximum UTF-16 length 8192. |
{
"input": {
"tokenText": "▁hello",
"suppliedBoundaryMarker": "▁"
},
"output": {
"hasMarker": true,
"unmarkedText": "hello"
}
}Limitations and assumptions
- The marker is literal and only one occurrence is removed. No tokenizer, word boundary, byte encoding or subword model is inferred.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Alternative or Complementary methods
Validate the complete corpus format, tokenizer, annotation sequence and scoring convention. Use a full parser or model evaluation when needed, preserving the exact offset unit, vocabulary and log base rather than inferring them from a local transformation.
References
Similar or alternative tools
- IOB Entity Tag Prefix Split
Split a supplied single IOB tag into prefix and entity-type text.
- CoNLL-U Decimal Empty Node Key
Split a supplied CoNLL-U-style decimal empty-node ID.