Description
Return useful and padding cache bytes.
Inference budgets depend on the number of stored values, sequence lengths, routing assignments and the chosen accounting basis. Logical cache bytes and padded token counts differ from actual device allocation, latency and billed usage. Caller-supplied prices, cache fractions and capacity assumptions must match the intended system.
Return useful and padding cache bytes.
When to use this tool
Use Kv Cache Head Padding Budget when reconciling the stated cache, padding, routing or price basis.
How to use the tool and how it works
- Enter heads, headDimension, paddedDimension, tokens, bytesPerElement, kvCopies. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read usefulBytes, paddingBytes. Use the output convention below.
Return useful and padding cache bytes. Require positive whole heads/headDimension/bytesPerElement/kvCopies, paddedDimension >= headDimension and tokens >= 0; useful multiplies true dimension, padding multiplies padded-minus-true dimension.1
| Input field | Meaning and units | Accepted input |
|---|---|---|
| heads | Heads | Required. Safe integer; minimum 1. |
| headDimension | Head Dimension | Required. Safe integer; minimum 1. |
| paddedDimension | Padded Dimension | Required. Safe integer. |
| tokens | Tokens | Required. Safe integer; minimum 0. |
| bytesPerElement | Bytes Per Element | Required. Safe integer; minimum 1. |
| kvCopies | Explicit supplied number of copies; neither a key/value factor nor layer count is inserted automatically. | Required. Safe integer; minimum 1. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| usefulBytes | Useful Bytes | Required. Safe integer. |
| paddingBytes | Padding Bytes | Required. Safe integer. |
{
"input": {
"heads": 8,
"headDimension": 64,
"paddedDimension": 80,
"tokens": 100,
"bytesPerElement": 2,
"kvCopies": 2
},
"output": {
"usefulBytes": 204800,
"paddingBytes": 51200
}
}Limitations and assumptions
- kvCopies is supplied rather than automatically two; no layer/batch multiplier or allocator overhead is added implicitly.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Use actual model/runtime layouts, allocator measurements and current provider billing records for real memory, performance or cost planning.
References
Similar or alternative tools
- Artificial Intelligence: Kv Cache Token Byte Count
Return an exact unpadded key/value-cache byte count.
- Inference Batch Sequence Waste Term
Return ragged-batch token totals and padding waste.