Description
Return ragged-batch token totals and padding waste.
Inference budgets depend on the number of stored values, sequence lengths, routing assignments and the chosen accounting basis. Logical cache bytes and padded token counts differ from actual device allocation, latency and billed usage. Caller-supplied prices, cache fractions and capacity assumptions must match the intended system.
Return ragged-batch token totals and padding waste.
When to use this tool
Use Inference Batch Sequence Waste Term when reconciling the stated cache, padding, routing or price basis.
How to use the tool and how it works
- Enter promptLengths, generatedLengths. Follow any array-alignment requirements below. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read liveTokens, paddedTokens, wastedTokens. Use the output convention below.
Return ragged-batch token totals and padding waste. Require aligned nonempty nonnegative whole prompt/generated lengths; live sums both lengths, padded = batch count*maximum combined length, wasted = padded-live.1
| Input field | Meaning and units | Accepted input |
|---|---|---|
| promptLengths | Nonnegative whole token counts aligned with generatedLengths. | Required. Array of safe integer; maximum items 256. |
| generatedLengths | Generated Lengths | Required. Array of safe integer; maximum items 256. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| liveTokens | Live Tokens | Required. Safe integer. |
| paddedTokens | Batch size times largest paired prompt-plus-generated length. | Required. Safe integer. |
| wastedTokens | Wasted Tokens | Required. Safe integer. |
{
"input": {
"promptLengths": [
3,
5
],
"generatedLengths": [
2,
3
]
},
"output": {
"liveTokens": 13,
"paddedTokens": 16,
"wastedTokens": 3
}
}Limitations and assumptions
- This assumes every sequence pads to the longest combined length; no tokenizer, dynamic cache allocation or actual speed/memory measurement is performed.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
Alternative or Complementary methods
Use actual model/runtime layouts, allocator measurements and current provider billing records for real memory, performance or cost planning.
References
Similar or alternative tools
- Kv Cache Head Padding Budget
Return useful and padding cache bytes.
- Artificial Intelligence: Kv Cache Token Byte Count
Return an exact unpadded key/value-cache byte count.