Whole Token Literal Replacer
Replace whole tokens under an explicit ASCII letter/digit/underscore boundary rule; Unicode word segmentation is not inferred.
Description
Replace whole tokens under an explicit ASCII letter/digit/underscore boundary rule; Unicode word segmentation is not inferred.
Whole-token replacement needs a defined token alphabet. This tool uses ASCII letters, digits and underscore to decide whether a literal match is embedded in a larger token.
When to use this tool
Replace whole tokens under an explicit ASCII letter/digit/underscore boundary rule; Unicode word segmentation is not inferred.
- Prepare the supplied text, needle, replacement fields for this operation.
- Compare the complete example with the representation used by your own input before carrying the result into another step.
How to use the tool and how it works
- Enter text, needle, replacement. Use the exact text syntax, units and index conventions explained below.
- Run the tool. If input validation fails, correct the supplied fields before interpreting any result.
- Read result.text, result.matchOffsets. Interpret false, null and empty lists using the distinctions below; they can be valid outputs.
Find nonoverlapping literal needle occurrences. Keep those with no ASCII token character immediately before or after, then replace retained spans from right to left and return original match offsets.1
| Input field | Meaning and representation | Accepted input |
|---|---|---|
| text | text. | Text; at most 8,192 UTF-16 code units, subject to the grammar above |
| needle | needle. | Text; at most 8,192 UTF-16 code units, subject to the grammar above |
| replacement | replacement. | Text; at most 8,192 UTF-16 code units, subject to the grammar above |
| Output field | Interpretation |
|---|---|
| result | Object containing the named outputs below. |
| result.text | The text value under the stated formula or text convention. |
| result.matchOffsets | The match offsets list in the ordering and representation described above. |
{
"input": {
"text": "cat cats cat",
"needle": "cat",
"replacement": "dog"
},
"output": {
"result": {
"text": "dog cats dog",
"matchOffsets": [
0,
9
]
}
}
}Limitations and assumptions
- The boundary rule is not Unicode word segmentation and need not match a programming language's identifiers. Empty needles are rejected; replacement text is literal and not rescanned.
- Text offsets and lengths use UTF-16 code units unless explicitly described as bytes or code points. CRLF, CR and LF are recognized as line endings; a terminal newline creates a final empty line in the shared line model.
- Every declared top-level field is required and unexpected fields are rejected. Numeric inputs must be finite; ordinary text fields are bounded to 8,192 UTF-16 code units, with tighter operation-specific limits where stated.
- Ordinary string lists have at most 128 entries of at most 128 code units each; numeric lists and matrices have separate bounded size checks. Nonfinite results and excessively large serialized outputs are rejected.
- Indexes are zero-based and spans have an exclusive end unless explicitly stated otherwise. A valid false, null or empty-list result is distinct from malformed input.
Alternative or Complementary methods
Use a parser or language-aware editor when quoting, syntax or nested constructs matter. For plain text transforms, keep UTF-16 offsets, Unicode code points, grapheme clusters and UTF-8 byte offsets distinct; apply changes to the same source version used to calculate spans.
References
Similar or alternative tools
- Literal Token Spacing Audit
Audit ASCII-space runs around every overlapping literal token; document edges pass, tabs are not counted as spaces.
- Enclosing Literal Wrapper Stripper
Strip one complete nonoverlapping enclosing literal wrapper, never internal or partial matches.