5 LLM APIs Tested for Latency: Real Data [2026]

Originally published at features that serve real users, I've learned the hard way that the only numbers worth trusting are the ones you measure yourself. So I did.

LLM API Latency Tested: 5 Models, 3 Prompt Sizes, Real Numbers

The benchmark I ran in March 2026 was deliberately simple. Five models, three prompt sizes (short ~50 tokens, medium ~200 tokens, long ~500 tokens), three runs each, all using streaming APIs to capture accurate TTFT. I disabled Gemini's thinking mode for a fair comparison. Every test ran from the same Toronto-based server, same network, same time window.

Claude Haiku 4.5 delivered its first token in 597 milliseconds on a medium-length prompt. GPT-4.1 Mini took roughly 4x longer on the identical test. That's not a rounding error. That's the difference between a user who stays engaged and one who's already reaching for the back button.

For developers building foundational research on response times showed that anything under 1 second keeps a user's flow of thought uninterrupted. Over 1 second, they start noticing the delay. Over 10 seconds, they leave.

The Full Comparison Table: TTFT, Throughput, and Cost

Every model I tested, side by side. I'm combining my first-party benchmark data with independent numbers from . Its TTFT is also the lowest I tested. If your priority is getting text on screen fast, nothing here touches it.

GPT-4.1 Mini is the biggest letdown. OpenAI markets it as their "lightweight" variant, but it's rated "slower than average" at 94.5 tokens/sec (#32/85). Not the cheapest. Not the fastest. Not great at anything in particular. OpenAI has already positioned GPT-5 Mini as the successor, which tells you everything.

GPT-4.1's throughput is legitimately strong at 125 tokens/sec (#13/85), putting it in the top quartile. But you're paying $2.00/$8.00 per million tokens for the privilege. That's 5x more than Gemini 2.5 Flash on input and 3.2x on output.

Which LLM API Is Fastest for Streaming?

I get this question constantly, and the honest answer is: it depends on what "fastest" means to you.

If you mean fastest first token — the thing that matters for chat UIs, systems: the model with the best benchmark throughput isn't always the model that finishes the job fastest. Verbosity is a hidden variable most benchmarks ignore.

Cost vs Latency: The Tradeoff Nobody Shows You

Most LLM latency comparisons pretend cost doesn't exist. That's wrong. In production, you don't optimize for speed in isolation. You optimize for speed at a cost you can sustain.

Say you're processing 10 million input tokens and 2 million output tokens per day. That's a moderate-traffic AI feature. Nothing crazy.

Daily API costs at list pricing:

• 
  Gemini 2.5 Flash: $3.00 input + $5.00 output = $8.00/day
  


• 
  GPT-4.1 Mini: $4.00 input + $3.20 output = $7.20/day
  


• 
  Claude Haiku 4.5: $8.00 input + $8.00 output = $16.00/day
  


• 
  GPT-4.1: $20.00 input + $16.00 output = $36.00/day
  


• 
  Claude Sonnet 4: $30.00 input + $30.00 output = $60.00/day
  

Gemini 2.5 Flash gives you the best speed and the second-lowest cost. GPT-4.1 Mini is cheapest but slowest. Claude Sonnet 4 costs 7.5x more than Gemini Flash for significantly less speed.

But factor in verbosity. If Gemini 2.5 Flash produces 1.8x more output tokens than GPT-4.1 Mini for the same task, your actual output token costs jump to $9.00 instead of $5.00. The "cheap" model isn't as cheap as the pricing page suggests.

This is why I always tell teams: run your actual prompts through multiple models and measure real output lengths before committing to one. The pricing page is a starting point, not the answer. If you're trying to manage these costs at scale, Netflix's patterns and . Many of the architectural tradeoffs apply here too.

Reliability and Uptime: The Metric Benchmarks Miss

Here's something I almost never see in latency comparison posts: uptime and reliability. A model that delivers 200ms TTFT 95% of the time but throws 503 errors the other 5% has worse effective latency than one delivering 600ms TTFT at 99.9% uptime.

I've been running . That's bigger than the TTFT gap between some of these models.

Which LLM API Should I Pick? A Decision Framework

After running these benchmarks and building production features on all five models, here's how I'd think about it:

User-facing chat with streaming, cost-sensitive: Gemini 2.5 Flash. Best TTFT, best throughput, lowest cost. Accept the verbosity and use max_tokens aggressively to rein it in.

User-facing chat with streaming, quality-sensitive: Claude Haiku 4.5. Costs a bit more, but you get more concise outputs and better instruction-following. Sub-600ms TTFT keeps users happy.

Backend processing, structured output, routing: GPT-4.1 Mini. The slow TTFT doesn't matter when nobody's watching a cursor blink. Very concise outputs keep costs low. Decent enough intelligence for classification.

Complex multi-step tasks, code generation: GPT-4.1 or Claude Sonnet 4. Higher intelligence, reasonable throughput. The cost premium pays for itself when accuracy saves you from expensive retries.

Maximum intelligence, latency is irrelevant: Claude Fable 5 or GPT-5.5. Batch processing, complex analysis, research tasks. These are not real-time models.

Best combined value across speed, intelligence, and cost: Qwen3.7 Max. 193 tok/s, 2.59s TTFT, 57 intelligence score, $1.43/M tokens blended. The catch: less ecosystem support than the big three, and you need to evaluate whether Alibaba's API reliability meets your bar.

One thing I want to stress for teams building calls in sequence, every millisecond multiplies. A four-step agent pipeline with 600ms TTFT per step adds 2.4 seconds just in first-token latency. At 2,400ms per step (GPT-4.1 Mini), you're at 9.6 seconds before the agent even starts producing useful output. TTFT should be your primary selection criterion for agent workloads. Not throughput. Not cost. TTFT.

For latency-sensitive production apps, pick the model with the best TTFT that meets your quality bar. Everything else — cost, throughput, intelligence — is negotiable. TTFT is what the user feels.

What the Benchmarks Still Can't Tell You

I want to be upfront about the limitations of this data. My benchmark uses three prompt sizes, three runs each, from a single geographic location. That's enough to establish relative ordering and ballpark numbers. It's not enough to make P99 guarantees.

alternatives to avoid API latency entirely, that's a different conversation — but for most production apps serving real users, the API route still wins on reliability and ease of deployment.

The LLM API world in late 2026 looks nothing like it did six months ago. GPT-5.5, Gemini 3.5 Flash, Claude Fable 5, and Qwen3.7 Max have all pushed the performance ceiling higher. But the fundamental tradeoff hasn't moved an inch: speed costs money, intelligence costs latency, and verbosity costs both. The developers who ship great products are the ones who know which of those three they can compromise on. And which one their users will never forgive them for getting wrong.

Frequently Asked Questions

Which LLM API has the lowest latency in 2026?

As of mid-2026, Gemini 2.5 Flash and Claude Haiku 4.5 deliver the lowest time-to-first-token among mainstream LLM APIs, both consistently under 600ms on medium-length prompts. For absolute lowest latency regardless of model size, North Mini Code and Gemini 2.5 Flash-Lite lead the Artificial Analysis leaderboard.

Is GPT-4.1 Mini actually fast?

No. Despite being OpenAI's lightweight model, GPT-4.1 Mini is rated "slower than average" at 94.5 tokens per second, ranking #32 out of 85 non-reasoning models. Its TTFT of approximately 2,400ms on medium prompts is roughly 4x slower than Claude Haiku 4.5. OpenAI has since positioned GPT-5 Mini as its successor.

How does Gemini 2.5 Flash verbosity affect real-world latency?

Gemini 2.5 Flash generates nearly 2x the average output tokens for the same task (17M tokens vs 9.2M average on the AA Intelligence Index). While its per-token speed is exceptional at 204.5 tok/s, the extra tokens inflate both end-to-end response time and output costs. Setting aggressive max_tokens limits can mitigate this.

What is a good time-to-first-token (TTFT) for a production LLM app?

For user-facing streaming applications, aim for under 1 second TTFT. Research on response time psychology shows users maintain their flow of thought below 1 second, start noticing delays between 1-10 seconds, and abandon after 10 seconds. Sub-600ms TTFT is the gold standard for chat interfaces.

Is it cheaper to run a local LLM instead of using an API?

It depends on your volume. At low to moderate usage (under a few million tokens per day), API costs are lower than the hardware amortization of running models locally. At high volume or for privacy-sensitive workloads, self-hosted inference on dedicated GPUs can be significantly cheaper per token. The break-even point varies by model size and hardware.

How do newer models like GPT-5.5 and Gemini 3.5 Flash compare on latency?

Both are significantly smarter than their predecessors but come with much higher latency. GPT-5.5 (high) has a 27.9s TTFT, and Gemini 3.5 Flash has 15-18s TTFT. These are reasoning-optimized models unsuitable for real-time streaming. For latency-sensitive work, the previous generation (GPT-4.1, Gemini 2.5 Flash) or their direct successors remain better choices.

Originally published on kunalganglani.com

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen