How Much Does It Cost to Self-Host Open Models on AWS?

Your AI bill tripled last quarter. Your CTO forwarded you an article about companies saving 70% by switching to open models. Now someone is asking you to figure out what that would actually look like.

I spent the last few weeks digging into this. The numbers, the hardware, the real trade-offs. Here's what I found, with enough specifics that you can actually make a decision rather than just nodding along to another "open source is the future" think piece.

What "Open Models" Actually Means

When someone says "open model" they mean an AI model where the weights (the learned parameters that make the model work) are publicly downloadable. You grab the file, run it on your hardware, and you don't pay anyone per request.

The big names right now: Meta's Llama 4, DeepSeek V4, Zhipu's GLM-5.2, Moonshot's Kimi K3, Alibaba's Qwen 3.5, and Google's Gemma 4.

These aren't toys. Some of them genuinely compete with the frontier models on real benchmarks. Chinese open models now handle over 30% of enterprise traffic on OpenRouter, up from 4.5% in early 2025. That's a massive shift in barely a year.

The Architecture: What You Actually Need

You want your team to use an open model. Here's the stack, from bottom to top.

Hardware (The Expensive Part)

A model is a giant file. We're talking anywhere from 4 GB (a small 7B model, quantized) to 1.5 TB (Kimi K3, full weights). That entire file needs to sit in GPU memory to run fast.

Why GPU memory specifically? Because generating each word in a response requires billions of multiply-and-add operations. GPUs do thousands of these in parallel. A CPU does them one at a time.

The practical difference: a 7B model on a CPU generates 2-5 tokens per second (painfully slow for interactive use). The same model on a GPU generates 30-80 tokens per second (feels instant). For one person on a CPU, it might be tolerable. For a team of 10 all hitting the same endpoint? Unusable. Requests queue up and everyone waits 30-60 seconds for responses.

Think of it like a highway. A CPU is a single lane with a high speed limit. A GPU is 4,000 lanes at a moderate speed. Language model inference is a traffic problem, not a speed problem. You need lanes, not a faster car.

The Serving Software (The Free Part)

Good news: the software stack is mature, open-source, and works today. No custom code required.

• 
  for a ChatGPT-like browser interface. User accounts, conversation history, file uploads. Your team won't know the difference from a commercial product.


• nginx or Caddy in front for authentication, TLS termination, and rate limiting.

The setup: install vLLM, run vllm serve meta-llama/Llama-4-Maverick, point Open WebUI at it, hand your team the URL. A day of work for someone comfortable with Linux. The vLLM API is OpenAI-compatible, which means any tool, extension, or script that works with the OpenAI API works here with zero code changes. Just swap the endpoint URL.

The Model

Download from : someone (or a small team) needs to keep this running. Model updates, instance reboots, monitoring, scaling adjustments. At 500 users that's justified. At 10, it might be more hassle than it's worth.

(Pricing based on AWS on-demand rates, August 2026. Check the . The infrastructure: a p6-b300 instance (8x NVIDIA B300 Blackwell Ultra GPUs) using vLLM as the serving engine, with Flexible Training Plans or Capacity Blocks for GPU reservation. Enterprise-grade, not a weekend project, but at least the path is documented.