Stop paying for idle GPUs in your CI: batching LLM eval jobs

TL;DR: Running LLM evaluations on every PR will burn your GPU budget faster than you can blink. We cut our eval spend by about 60% by batching jobs into windowed runs on shared GPU pools, plus a smarter queue that knows the difference between a "smoke test" eval and a full regression run. Here's how, and where the trade-offs hurt.

Right, so a few months back I got pulled into a conversation that's becoming pretty familiar around here. A team had wired up an LLM-based evaluation suite into their CI. Every PR triggered a run against a set of prompts, scored the outputs, and posted results back to the PR. Lovely in theory.

The cloud bill was not lovely.

They were spinning up a g5.xlarge per PR, sometimes three or four in parallel during peak hours, and the GPU sat idle for about 70% of the run because most of the time was spent on cold starts, model loading, and prompt formatting. Classic case of treating GPUs like CPUs.

I reckon a lot of teams are hitting this wall right now. So let's talk about what actually works.

The problem with "GPU per job"

CI runners are designed for stateless, throwaway compute. That model breaks the second you involve a 7B+ parameter model that takes 30-90 seconds to load into VRAM.

Here's the rough breakdown of a typical eval job we measured:

So out of about 3 minutes of billable GPU time, you're getting 40 seconds of useful work. That's brutal economics.

Batching: the boring fix that works

The trick isn't fancy. You stop spinning up a GPU per job and start treating the GPU like a long-lived service that consumes jobs from a queue.

We run a small pool of g5.xlarge instances (usually 2-4 depending on load) that stay warm. Each runner has the model preloaded in VRAM. CI jobs push eval requests to an SQS queue, runners pull from the queue, batch up to N prompts per inference pass, and post results back.

Rough sketch of the runner config:

runner:

  instance_type: g5.xlarge

  pool_size_min: 2

  pool_size_max: 6

  scale_metric: queue_depth

  scale_threshold: 25  # jobs in queue



  model:

    name: llama-3.1-8b-instruct

    preload: true

    keep_warm_seconds: 1800



  batching:

    max_batch_size: 16

    max_wait_ms: 2000



  job_types:

    smoke_eval:

      priority: high

      max_prompts: 10

    full_regression:

      priority: low

      max_prompts: 500

      window: nightly_only

That max_wait_ms is doing the heavy lifting. The runner waits up to 2 seconds to gather a batch before firing inference. For CI, 2 seconds of latency is nothing. For inference throughput, it's everything.

Routing matters too

Once you've got a warm pool, you might as well route different model calls through one place. We have eval suites that hit a mix of self-hosted Llama, Claude via API, and OpenAI. Instead of each CI job authenticating separately and managing keys, we put a gateway in front.

There's a bunch of options here. LiteLLM is popular, Bifrost ( - the AWS docs are surprisingly readable on this

No worries if your setup looks different. The general shape holds: warm pools, batched jobs, classified workloads. Apply where it fits.