Which Metrics Matter in an LLM API Benchmark?
Compare P50/P95/P99, TTFT, success rate, 429 limits, throughput, prompt cache, model purity and actual token cost in an LLM API benchmark.
Quick Answer
A useful LLM API benchmark combines success rate, P95 latency, TTFT, output speed, 429 behavior, prompt cache, model purity and actual token cost. Average latency or a single speed test cannot represent production experience, and comparisons must fix model, region, prompt and output length.
Key Takeaways
- Pair average latency with P95/P99 to expose tail behavior.
- Chat UX depends on TTFT; batch jobs emphasize completion time.
- Speed rankings require acceptable model purity and success rate first.
- Use actual usage and cache hits to calculate cost.
Group metrics into four dimensions
A procurement-grade benchmark covers reliability, response experience, protocol quality and economics. A speed-only test rewards channels that fail often or silently route to a smaller model.
| Dimension | Core metrics | Question answered |
|---|---|---|
| Reliability | Success rate, 5xx, 429, timeout | Will requests finish consistently? |
| Experience | TTFT, P50/P95/P99, output tokens/s | How long does a user wait? |
| Protocol quality | Model purity, stream, tools, usage, cache | Is the API complete and undowngraded? |
| Economics | Input, output and cache token cost | What does one task really cost? |
Using P50, P95, P99 and the average
P50 describes a typical request, P95 captures slow-tail experience, and P99 exposes extreme outliers. An average can be distorted by a few failures or hide a bimodal distribution, so always publish sample size with percentiles.
TTFT and output speed measure different things
TTFT is the delay before the first meaningful content chunk. Output tokens per second describes sustained generation afterward. A channel can start quickly and generate slowly, or do the reverse.
Streaming checks should also track empty chunks, chunk gaps, disconnects and whether final usage is returned correctly.
Reliability and rate limits come before speed
Count 2xx, authentication failures, 429, 5xx and timeouts separately. Combine 429 responses with rate-limit headers, retry delays and concurrency boundaries. Report first-attempt success as well as success after retries.
Model purity, cache and real cost
Smaller or quantized models are often faster and cheaper, so identity and protocol consistency must be a ranking gate. Otherwise a benchmark rewards silent downgrade.
Calculate cost from actual input, output and cache usage. For long-context workloads, cache hit rate can matter more than list price.
Making providers comparable
Fix model version, prompt, temperature, max tokens, streaming mode, region and concurrency. Warm up before sampling, test across time windows, and disclose sample size, failure handling and pricing date.
- Chat: prioritize TTFT, P95, stream stability and success rate.
- Agents: prioritize tool calls, tail latency, 429 and retry cost.
- Batch: prioritize throughput, P99, unit task cost and recovery.
- Long context: prioritize cache hit, input cost and context limits.
Frequently Asked Questions
How many requests make an LLM API benchmark meaningful?
Small samples are useful for screening. Procurement and SLA decisions need larger samples across time windows, with sample size disclosed. P95/P99 are unstable with too few observations.
Is TTFT or total latency more important?
Chat products emphasize TTFT and stream stability; batch workloads emphasize completion time and throughput. Weight metrics by workload.
Why not compare official list prices only?
Actual cost also depends on output length, cache hits, failed retries and model downgrade. Calculate cost per completed task from usage.
Continue Reading
Run a Check with a Real API
Use a temporary key to inspect connectivity, latency, cache, rate limits, model purity and token cost.