Testing LLM Speed Across Cloud Providers: Groq, Cerebras, AWS & More

After to benchmark different LLM providers consistently. The framework:

• Runs a series of standardised prompts across different providers


• Measures inference time and response generation


• Writes results to structured output files


• Supports multiple providers including OpenAI, Anthropic, AWS Bedrock, Groq, and Cerebras

The test prompts were designed to cover different scenarios:

• Mathematical computations (typically challenging for LLMs)


• Long-form text summarisation (high input tokens, lower output)


• Structured output generation (JSON, XML, CSV formats)

Test Results

The complete benchmark results are available in

One of the most interesting findings was the significant speed variation for identical models across different providers. This suggests that infrastructure and optimization play a crucial role in inference speed.

Groq's performance across different model sizes reveals an intriguing pattern: whether running small or large models, inference speeds remain remarkably consistent, suggesting they possibly managed to optimise for bigger models.

Key Findings

• 
  Groq and Cerebras: The hype is real. Both providers demonstrated exceptional performance, particularly with larger models like Llama 3 70B


• 
  Ollama: With a decent GPU (e.g., RTX 4090), smaller models (Llama 3.2 1B/3B) performed (speed-wise) comparably to the quickest "API-based models" like Anthropic's Claude Haiku 3 and Amazon's Nova Micro


• 
  Speed rankings were fairly consistent across different prompts (math, summarisation, structured output)


• 
  API throttling became an issue with larger models on AWS Bedrock (Claude Sonnet 3.5, Opus 3, Nova Pro)