Alibi Attention One Distance Penalty Tool

Description

Return one ALiBi-style distance penalty.

Attention calculations distinguish query/key scores, head layouts and visibility masks. A dot-product term, head grouping or supplied Boolean mask describes one part of a model; it does not run attention, infer a positional schedule or establish the behavior of a complete implementation.

Return one ALiBi-style distance penalty.

When to use this tool

Use Alibi Attention One Distance Penalty when checking the stated attention coordinate, visibility, angle or head-layout convention.

How to use the tool and how it works

Enter distance, slope, baseLogit. Use the units and conventions below.
Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
Read bias, biasedLogit. Use the output convention below.

Return one ALiBi-style distance penalty. Require nonnegative whole distance and slope >= 0; bias = -slope*distance, biasedLogit = baseLogit+bias.^[1]

ALiBi applies distance-proportional attention penalties. This helper uses the supplied slope without generating the paper’s head-specific slopes or running attention.^[2]

Input field	Meaning and units	Accepted input
distance	Distance	Required. Safe integer; minimum 0.
slope	Supplied nonnegative distance-penalty slope; no ALiBi slope schedule is generated.	Required. Finite number; minimum 0.
baseLogit	Base Logit	Required. Finite number.

Output field	Interpretation	Returned shape
bias	Bias	Required. Finite number.
biasedLogit	Biased Logit	Required. Finite number.

json

{
  "input": {
    "distance": 5,
    "slope": 0.2,
    "baseLogit": 3
  },
  "output": {
    "bias": -1,
    "biasedLogit": 2
  }
}

Limitations and assumptions

The slope and distance convention are supplied; no head-slope schedule, causal mask or full attention model is constructed.
Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.

Alternative or Complementary methods

Inspect the complete model configuration and implementation for positional schedules, masking, softmax and actual inference behavior.

References

Calculation and input contract: artificial-intelligence/alibi-attention-one-distance-penalty — tool.io
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — Ofir Press, Noah A. Smith and Mike Lewis

Similar or alternative tools

Attention Query Key Dot Scale Term
Return scaled query-key dot product.
Rotary Angle Wrap Token
Wrap a supplied rotary angle to one turn.