⚡ LLM Tokens per Second Calculator
Calculate LLM generation throughput, requests per minute, GPU utilization, and capacity from tokens, latency, batch size, and concurrent replicas.
Calculate LLM Tokens per Second
What this LLM tokens per second calculator calculates
Convert observed generation time into token throughput and estimate request capacity under a chosen concurrency and utilization target.
Separate time to first token from generation speed. Users experience both prompt processing delay and streaming decode rate.
LLM Tokens per Second Calculator formula
Assumptions and limitations
The result is an engineering estimate based on the values entered. AI models, tokenizers, runtimes, accelerators, cloud services, and provider billing rules differ. Validate important decisions with measured data from the exact model, hardware, framework, and pricing plan you intend to use.
Worked example
Generating 500 tokens in 10 seconds equals 50 tokens per second. Four replicas at 80% utilization can complete about 19.2 ten-second requests per minute at batch one.
How to use the result
Separate time to first token from generation speed. Users experience both prompt processing delay and streaming decode rate.
- Start with representative production assumptions rather than best-case demos.
- Run a low, expected, and high scenario to understand the range.
- Record model version, pricing date, hardware, precision, and workload details.
- Replace assumptions with observed p50 and p95 measurements after testing.
Common mistakes to avoid
- Combining prompt prefill and decode into one throughput number without labeling it.
- Multiplying single-request throughput by concurrency without accounting for contention.
- Sizing at 100% utilization, leaving no room for bursts or tail latency.
Frequently asked questions
What is time to first token?
It is the delay before streaming output begins, often dominated by prompt processing and queue time.
Why can batching improve throughput but hurt latency?
Batching uses hardware more efficiently but may add waiting time and reduce per-user responsiveness.
Should I plan from average or p95 latency?
Use p95 or another service-level percentile for capacity and user-experience planning.
Related AI calculators
Methodology and privacy
This educational tool uses the formula and assumptions displayed on the page. Calculations run locally in your browser, and the page does not transmit the values you enter. Results are estimates rather than provider quotes, benchmark guarantees, financial advice, or capacity guarantees. Last methodology review: August 2, 2026.
Formula Explorer connections
Interpretation: This formula describes how token load, memory and throughput constrain an online model deployment. Assumption: Tokenizer, model architecture, batching, queueing, runtime, hardware and traffic distribution must match the intended production system.