🧠 LLM KV Cache Memory Calculator
Estimate LLM KV cache memory per token, request, and concurrent batch using layers, KV heads, head dimension, precision, and tensor parallelism.
Calculate LLM KV Cache Memory
What this LLM KV cache memory calculator calculates
Autoregressive inference stores attention keys and values for prior tokens so the model does not recompute the full sequence at every decoding step. The KV cache grows with sequence length and concurrent requests and can become the limiting resource after model weights fit in memory.
Grouped-query and multi-query attention reduce KV memory because the number of key-value heads can be smaller than the number of query heads. Cache precision, tensor parallelism, paged allocation, sliding windows, and hybrid model architectures also affect the real allocation.
LLM KV Cache Memory Calculator formula
Assumptions and limitations
The result is an engineering estimate based on the values entered. Model architectures, tokenizers, frameworks, accelerators, parallel strategies, billing rules, and production traffic differ. Validate important decisions with measured data from the exact model, runtime, hardware, and workflow you intend to use.
Worked example
A 32-layer model with 8 KV heads, 128 dimensions, FP16 cache, 8,192 tokens, and 16 concurrent sequences uses roughly 16 GiB before allocator overhead on one rank. FP8 cache can approximately halve the raw element storage when supported.
How to interpret and use the result
Use the per-request result to estimate how context length competes with concurrency. Reserve memory for model weights, kernels, CUDA graphs, activations, and framework overhead before assigning the remainder to KV cache. Benchmark the exact model because sliding-window, state-space, cross-attention, and hybrid layers may not follow the uniform formula.
- Start with measured or representative production assumptions.
- Run conservative, expected, and optimistic scenarios.
- Record model version, framework, precision, hardware, and review date.
- Replace estimates with observed p50 and p95 values after testing.
Common mistakes to avoid
- Using query-head count instead of key-value-head count for GQA or MQA models.
- Sizing only for prompt tokens and forgetting generated tokens also remain in cache.
- Assuming all GPU memory is available after loading model weights.
Methodology references
These sources describe the underlying technology. The calculator remains an approximation because implementations and workloads differ.
Frequently asked questions
What is the difference between KV cache and model weights?
Weights are persistent model parameters; KV cache is request-dependent state that grows with active tokens and sequences.
Can FP8 reduce KV cache memory?
Yes, supported runtimes can store KV cache in FP8, which reduces element size, though quality and backend support must be validated.
Why does vLLM report a different value?
Paged allocation, hybrid layer groups, block rounding, offloading, and implementation-specific metadata can change practical capacity.
Related AI calculators
Methodology and privacy
This educational tool uses the formula and assumptions displayed on the page. Calculations run locally in your browser, and the page does not transmit the values you enter. Results are estimates rather than provider quotes, benchmark guarantees, financial advice, or capacity guarantees. Last methodology review: August 2, 2026.
Formula Explorer connections
Interpretation: KV cache stores attention keys and values for every cached token, layer and active sequence. Memory therefore grows linearly with context and concurrency. Assumption: the estimate depends on the model’s KV-head architecture, precision, parallel sharding and runtime overhead.