If you've been using ray-project/llmperf, you may have noticed it's now in archive mode. No new updates, no fixes, no responses to issues. If you're evaluating it for the first time, that's worth knowing before you build anything on top of it.
This page is part of my LLM benchmarking guide, which covers the metrics themselves (TTFT, ITL, throughput). Here I want to focus on the tools — what's out there now that llmperf is effectively done, and what I ended up building.
What llmperf was good at
Credit where it's due — llmperf was the go-to open-source option for benchmarking OpenAI-compatible endpoints. It measured the metrics that matter (TTFT, ITL, throughput), handled concurrency, and came out of the Anyscale/Ray team, so it had credibility. For a lot of teams it did the job well.
Why I started looking around
Two things pushed me to look at alternatives, and both are about fit rather than flaws:
ITL aggregation. llmperf computes Inter-Token Latency by averaging within each request first, then aggregating those per-request averages. That's a reasonable choice and works well for many use cases. But I was specifically trying to catch latency spikes during the decode phase, and per-request averaging smooths exactly those out. I needed the raw distribution.
Startup overhead. Runs spawn Ray workers, so there's a meaningful spin-up cost before the first request fires. When I just want to poke at an endpoint quickly, that's more ceremony than I want — I was after something closer to curl than a cluster setup.
Neither is a flaw. They're design decisions that matched llmperf's goals and didn't match mine.
The alternatives
| Tool | Single binary / low deps | GPU-level metrics | Distributed | Notes |
|---|---|---|---|---|
| aiperf | No | Yes | Yes | Python package, very comprehensive |
| vllm-bench | No | Yes | No | vLLM-specific |
| trtllm-bench | No | Yes | No | TensorRT-LLM specific |
| GuideLLM | No | Partial | No | Strong reporting and dashboards |
| llmperf-rs | Yes | No | No | Single Rust binary |
There's also a note for genai-perf: NVIDIA sunsetted it and moved development to aiperf.
When to use which
You need GPU-level metrics (prefix caching, kernel-level breakdown) → aiperf or trtllm-bench. This is where llmperf-rs won't help you.
You're testing vLLM-specific behavior → vllm-bench.
You want dashboards and visual reporting → GuideLLM.
You need distributed load generation → aiperf.
You just want to hit an OpenAI-compatible endpoint quickly, with minimal setup, and see TTFT/ITL/throughput → llmperf-rs.
That last one is the gap I was trying to fill — something I could drop onto a box and run in seconds, that preserved raw ITL values so spikes weren't hidden.
What I built: llmperf-rs
llmperf-rs is a single Rust binary that benchmarks any OpenAI-compatible endpoint (vLLM, Ollama, local APIs). It:
- Keeps raw ITL values across all responses before computing percentiles, so p99 and max actually reflect spikes.
- Uses API-reported token counts from the
usagefield when available, falling back to a tokenizer you specify. The original llmperf used one tokenizer for everything, which gets inaccurate across model families. - Outputs console summaries plus JSON for digging in with pandas.
It's not trying to compete with the GPU-deep tools. I think of it as one level above curl — fast to start, low dependency, good enough for most "how's this endpoint doing" questions.
If that trade-off sounds right for you: grab it from the releases page, or cargo install --git https://github.com/wheynelau/llmperf-rs.
The takeaway
llmperf being archived doesn't mean the tooling disappeared — aiperf, vllm-bench, GuideLLM, and others are all actively maintained. The choice mostly comes down to how deep you need to go (GPU metrics vs endpoint metrics) and how much setup you're willing to tolerate.
You can find this post and more on my blog.
SOCIAL SHARE CARD GENERATOR