Regex Replacement Tokenizer
Tokenize ECMAScript replacement-dollar syntax without applying matches; numeric reference resolution depends on separately supplied capture evidence.
Description
Tokenize ECMAScript replacement-dollar syntax without applying matches; numeric reference resolution depends on separately supplied capture evidence.
Replacement strings have token syntax distinct from regex pattern syntax. Tokenizing them exposes literal text and substitution markers.1
When to use this tool
Tokenize ECMAScript replacement-dollar syntax without applying matches; numeric reference resolution depends on separately supplied capture evidence.
- Prepare the supplied replacement fields for this operation.
- Compare the complete example with the representation used by your own input before carrying the result into another step.
How to use the tool and how it works
- Enter replacement. Use the exact text syntax, units and index conventions explained below.
- Run the tool. If input validation fails, correct the supplied fields before interpreting any result.
- Read result.tokens. Interpret false, null and empty lists using the distinctions below; they can be valid outputs.
Scan replacement left to right for dollar-dollar, whole-match, prefix, suffix, one-or-two-digit and named-reference tokens. Return each type, original spelling and zero-based UTF-16 position; group intervening text as literal tokens.2
| Input field | Meaning and representation | Accepted input |
|---|---|---|
| replacement | replacement. | Text; at most 8,192 UTF-16 code units, subject to the grammar above |
| Output field | Interpretation |
|---|---|
| result | Object containing the named outputs below. |
| result.tokens | The tokens list in the ordering and representation described above. |
{
"input": {
"replacement": "a$1$$"
},
"output": {
"result": {
"tokens": [
{
"type": "literal",
"text": "a",
"position": 0
},
{
"type": "numbered",
"text": "$1",
"position": 1
},
{
"type": "dollar",
"text": "$$",
"position": 3
}
]
}
}
}Limitations and assumptions
- Tokens are not expanded against a match. Their eventual meaning can depend on the actual captures; the tokenizer alone does not certify valid references.
- All declared input fields are required and unexpected top-level fields are rejected. Numbers must be finite; lists are limited to 128 entries and ordinary text fields to 8,192 UTF-16 code units, with tighter limits where stated.
- JSON text inputs must parse within bounded nesting and size limits. Numeric outputs outside the safe finite range are rejected; large exact integer results use decimal strings. Indexes and half-open spans are zero-based unless a rank is explicitly described as one-based.
Alternative or Complementary methods
Compile generated pattern sources with explicitly chosen flags and test them on bounded representative text. Pattern sources, replacement strings and JSON string spellings require different escaping conventions.
References
-
ECMAScript specification: GetSubstitution — Ecma International / TC39
-
Tool behavior and example: Regex Replacement Tokenizer — tool.io
Similar or alternative tools
- Regex Replacement Literal Escape
Escape every dollar sign for literal ECMAScript string replacement; unlike regex pattern escaping this preserves replacement text semantics.
- Regex Replacement Reference Audit
Audit replacement references against an explicit capture count/name bank, accounting for two-digit-to-one-digit fallback; unresolved references remain literal in ECMAScript, not execution errors.