Logit Mask Forbidden Token

Description

Remove supplied forbidden token IDs and their aligned logits.

Model and prompt records distinguish token indices, offset units, reserved capacity, code scales and numeric representations. A common softmax shift is mathematically invariant but can alter rounded inputs. Literal brace doubling and citation tokens are format conventions rather than universal template or evidence validation.

Remove supplied forbidden token IDs and their aligned logits.

When to use this tool

Use Logit Mask Forbidden Token when you already have tokenIds, logits, forbiddenTokenIds and need to remove supplied forbidden token IDs and their aligned logits.

How to use the tool and how it works

Enter tokenIds, logits, forbiddenTokenIds. Follow any array-alignment requirements below. Use the units and conventions below.
Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
Read allowedTokenIds, allowedLogits, originalIndices. Use the output convention below.

Require aligned tokenIds/logits and unique token IDs; retain indices whose IDs are absent from forbiddenTokenIds and return allowed IDs/logits in original order.^[1]

Input field	Meaning and units	Accepted input
tokenIds	Token Ids	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.
logits	Logits	Required. Array of finite number; maximum items 256.
forbiddenTokenIds	Forbidden Token Ids	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.

Output field	Interpretation	Returned shape
allowedTokenIds	Allowed Token Ids	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.
allowedLogits	Entries physically removed for forbidden IDs, with preserved order; no negative-infinity mask or normalization.	Required. Array of finite number; maximum items 256.
originalIndices	Original Indices	Required. Array of safe integer; maximum items 256.

json

{
  "input": {
    "tokenIds": [
      "a",
      "b",
      "c"
    ],
    "logits": [
      2,
      -1,
      3
    ],
    "forbiddenTokenIds": [
      "b"
    ]
  },
  "output": {
    "allowedTokenIds": [
      "a",
      "c"
    ],
    "allowedLogits": [
      2,
      3
    ],
    "originalIndices": [
      0,
      2
    ]
  }
}

Limitations and assumptions

This removes entries rather than inserting negative infinity. Empty results are allowed; no softmax, decoding, vocabulary verification or actual model constraint is performed.
Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.

Alternative or Complementary methods

Use the complete tokenizer, template, model contract and evaluation method. Verify actual capacity, vocabulary and numerical behavior when supplied budgets, local masks or representation checks cannot establish correct model operation.

References

Calculation and input contract: artificial-intelligence/logit-mask-forbidden-token — tool.io

Similar or alternative tools

Softmax Logit Shift Invariance Residual
Measure numerical softmax change after a supplied common logit shift.
Quantized Weight Zero Point Code
Center a supplied quantized code around its zero point.