Description
Calculate supplied token surprisal in bits.
Language-processing records distinguish tokens, code points, UTF-16 positions and UTF-8 byte offsets. Tag grammars and local transition checks do not validate a full annotation sequence. Log-loss units determine perplexity conversion, while identifier suffixes may be textual labels rather than ordinary decimal numbers.
Calculate supplied token surprisal in bits.
When to use this tool
Use Lexical Surprisal Token Bits when you already have tokenProbability and need to calculate supplied token surprisal in bits.
How to use the tool and how it works
- Enter tokenProbability. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read surprisalBits. Use the output convention below.
Require tokenProbability in (0,1]; return -log2(probability).1
Logarithm base sets the information unit. A negative base-two log of one supplied probability is an individual surprisal, not an empirical entropy estimate.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| tokenProbability | Token Probability | Required. Finite number; maximum 1; strictly above 0. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| surprisalBits | -log2(tokenProbability), for a supplied probability in (0,1]; zero is rejected. | Required. Finite number. |
{
"input": {
"tokenProbability": 0.25
},
"output": {
"surprisalBits": 2
}
}Limitations and assumptions
- Probability is supplied, not estimated or contextually verified. Zero probability is rejected rather than returning infinity; this is not an average entropy or full model score.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Validate the complete corpus format, tokenizer, annotation sequence and scoring convention. Use a full parser or model evaluation when needed, preserving the exact offset unit, vocabulary and log base rather than inferring them from a local transformation.
References
Similar or alternative tools
- Language Model Log Base Perplexity Crosswalk
Convert supplied mean negative log likelihood between bases and perplexity.
- Coreference Mention Number Label
Label a supplied document mention number and coreference cluster.