Description
Convert supplied mean negative log likelihood between bases and perplexity.
Language-processing records distinguish tokens, code points, UTF-16 positions and UTF-8 byte offsets. Tag grammars and local transition checks do not validate a full annotation sequence. Log-loss units determine perplexity conversion, while identifier suffixes may be textual labels rather than ordinary decimal numbers.
Convert supplied mean negative log likelihood between bases and perplexity.
When to use this tool
Use Language Model Log Base Perplexity Crosswalk when you already have meanNegativeLogLikelihood, logarithmBase and need to convert supplied mean negative log likelihood between bases and perplexity.
How to use the tool and how it works
- Enter meanNegativeLogLikelihood, logarithmBase. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read perplexity, natsPerToken, bitsPerToken. Use the output convention below.
Require meanNLL>=0 and logarithmBase>1; nats=meanNLL*ln(base), bits=nats/ln(2), perplexity=exp(nats).1
Information measures change scale with the logarithm base. Convert the supplied mean loss consistently before exponentiating; this does not evaluate token probabilities or a language model.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| meanNegativeLogLikelihood | Mean token loss measured in the supplied logarithm base, not necessarily nats. | Required. Finite number; minimum 0. |
| logarithmBase | Finite base greater than one. | Required. Finite number; strictly above 1. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| perplexity | exp(meanNegativeLogLikelihood*ln(logarithmBase)); tokenization and averaging must already be consistent. | Required. Finite number. |
| natsPerToken | Nats Per Token | Required. Finite number. |
| bitsPerToken | Bits Per Token | Required. Finite number. |
{
"input": {
"meanNegativeLogLikelihood": 3,
"logarithmBase": 2
},
"output": {
"perplexity": 7.999999999999998,
"natsPerToken": 2.0794415416798357,
"bitsPerToken": 3
}
}Limitations and assumptions
- The supplied mean must already use a consistent token and weighting basis. No language model, token probability, data split, comparable vocabulary or corpus score is computed.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Validate the complete corpus format, tokenizer, annotation sequence and scoring convention. Use a full parser or model evaluation when needed, preserving the exact offset unit, vocabulary and log base rather than inferring them from a local transformation.
References
Similar or alternative tools
- Lexical Surprisal Token Bits
Calculate supplied token surprisal in bits.
- Coreference Mention Number Label
Label a supplied document mention number and coreference cluster.