Description
Return one supplied policy/reference log-probability reward term.
A policy/reference probability ratio compares how strongly two distributions support the same supplied outcome. Its natural logarithm is positive above a ratio of one and negative below it. A scaled single-token ratio is only one contribution; it does not recover sequence probabilities, a complete reward or a preference-training objective.
Return one supplied policy/reference log-probability reward term.
When to use this tool
Use Reward Reference Log Ratio One Token when checking the stated probability, score, guidance or scale contribution.
How to use the tool and how it works
- Enter policyProbability, referenceProbability, beta. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read logRatio, scaledReward. Use the output convention below.
Return one supplied policy/reference log-probability reward term. Require both probabilities in (0,1] and beta >= 0; logRatio = ln(policy/reference), scaledReward = beta*logRatio.1
Preference optimization uses policy/reference log ratios within a complete sequence-level objective. This helper evaluates one supplied scaled ratio without constructing that objective or training a model.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| policyProbability | Policy Probability | Required. Finite number; maximum 1; strictly above 0. |
| referenceProbability | Reference Probability | Required. Finite number; maximum 1; strictly above 0. |
| beta | Beta | Required. Finite number; minimum 0. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| logRatio | Natural logarithm of policyProbability/referenceProbability, computed after the division. | Required. Finite number. |
| scaledReward | beta times logRatio; one contribution rather than a complete preference objective. | Required. Finite number. |
{
"input": {
"policyProbability": 0.4,
"referenceProbability": 0.2,
"beta": 2
},
"output": {
"logRatio": 0.6931471805599453,
"scaledReward": 1.3862943611198906
}
}Limitations and assumptions
- This is a single token contribution, not a fitted reward model, complete KL divergence or preference-training algorithm.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Use the full distribution and model-specific decoding or training objective for normalized probabilities, preference optimization or actual generation.
References
-
Calculation and input contract: artificial-intelligence/reward-reference-log-ratio-one-token — tool.io
-
Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafael Rafailov and coauthors
Similar or alternative tools
- Log Probability Nats Bits Crosswalk
Convert a nonpositive log probability from nats to bits.
- Logit Bias One Token Odds Factor
Return one-token logit-bias odds transformation.