The GPU Is the New Database

_

Twenty years ago, teams had no idea how to run databases at scale. They made every mistake possible before the patterns solidified. We are now in the same position with GPU infrastructure, making the same mistakes, faster.

Find more articles atThis site

_

In 2004, if you were running a web application at any meaningful scale,

your biggest infrastructure problem was the database. Not the application

servers, those were stateless, you could add more. The database was the

single stateful thing everything depended on, it didn't scale horizontally,

it was expensive to run, and almost nobody knew how to operate it well.

Teams made every mistake. They put too much logic in the application and

not enough in the database. They put too much in the database and not

enough in the application. They didn't index correctly. They didn't cache

correctly. They scaled vertically until they couldn't, then scrambled to

shard. They had no idea what their query plans looked like. They treated

the database as a black box until it stopped working, then learned the

hard way that it wasn't.

Over the following decade, the patterns solidified. Connection pooling.

Read replicas. Query analysis. Proper indexing strategy. Cache layers.

The knowledge became common. The tools improved. Managed database services

abstracted most of the complexity. Today a competent team can run a

database at significant scale without extraordinary expertise.

We are now, in 2026, in the same position with GPU infrastructure. The

GPU is the new database, the expensive, stateful, poorly-understood

bottleneck that everything AI depends on, that doesn't scale the way

people expect, that is being operated badly by the majority of teams

running it, and for which the patterns have not yet solidified.

The teams that figure this out first will have an infrastructure advantage

that is very difficult to close. The teams that don't will spend the

next five years making the same mistakes everyone made with databases

in 2004, just faster and more expensively.

Why the GPU is not just a fast CPU

The first mistake most teams make with GPU infrastructure is treating

GPUs as very fast CPUs. They're not. They're a fundamentally different

computational model, and the mismatch between that model and how most

people use them is where most of the waste comes from.

A CPU is optimised for latency ,completing a single complex task as

quickly as possible. It has a small number of powerful cores, large

caches, sophisticated branch prediction, and out-of-order execution.

It's good at sequential logic, conditional branching, and tasks where

each step depends on the result of the previous one.

A GPU is optimised for throughput, completing an enormous number of

simple tasks simultaneously. It has thousands of smaller, simpler cores.

It's good at the same operation applied in parallel to a large amount

of data. It's bad at anything sequential, anything with complex

branching, and anything where you need to move data back to the CPU

in the middle of computation.

The practical consequence: a GPU that is not batching work is a GPU

that is mostly idle. The most common pattern for teams deploying AI

inference in production, one request comes in, run the model, return

the result, wait for the next request, uses a small fraction of the

GPU's actual capacity. The GPU's utilisation number looks reasonable.

The GPU's actual computational throughput is terrible.

This is the equivalent of a database that opens a new connection for

every query, executes it, and closes the connection. Technically

functional. Completely missing how the system should be used.

# What most teams do: one request, one inference

# GPU utilisation looks like 20-40%, but throughput is poor



async def handle_inference_request(prompt: str) -> str:

    result = model.generate(prompt)  # GPU mostly idle while waiting

    return result





# What should be happening: dynamic batching

# Multiple requests grouped and processed together



class InferenceBatcher:

    def __init__(self, model, max_batch_size: int = 32, max_wait_ms: int = 50):

        self.model = model

        self.max_batch_size = max_batch_size

        self.max_wait_ms = max_wait_ms

        self.queue: asyncio.Queue = asyncio.Queue()



    async def infer(self, prompt: str) -> str:

        future = asyncio.Future()

        await self.queue.put((prompt, future))

        return await future



    async def _batch_worker(self):

        while True:

            batch = []

            deadline = asyncio.get_event_loop().time() + (self.max_wait_ms / 1000)



            # Collect requests until batch is full or deadline passes

            while len(batch) < self.max_batch_size:

                timeout = deadline - asyncio.get_event_loop().time()

                if timeout <= 0:

                    break

                try:

                    item = await asyncio.wait_for(

                        self.queue.get(),

                        timeout=timeout

                    )

                    batch.append(item)

                except asyncio.TimeoutError:

                    break



            if not batch:

                continue



            prompts = [item[0] for item in batch]

            futures = [item[1] for item in batch]



            # Single GPU call processes all requests simultaneously

            results = self.model.generate_batch(prompts)



            for future, result in zip(futures, results):

                future.set_result(result)

Dynamic batching is the connection pooling of GPU inference. It is

not optional if you care about cost or throughput. It is also not

implemented by default in most hand-rolled inference deployments,

for the same reason that early web applications didn't implement

connection pooling: teams didn't know they needed it until they

hit the wall.

The memory hierarchy nobody teaches you

GPU memory is not like CPU memory. Understanding the difference is

the difference between a system that works and one that doesn't, and

between inference costs that are manageable and ones that are not.

A GPU has its own on-device memory ,VRAM. VRAM is fast, finite,

and expensive. A GPU with 80GB of VRAM is a very expensive GPU.

The model you're running must fit in VRAM. If it doesn't fit, you

can use techniques like quantization to make it smaller, or you can

distribute it across multiple GPUs, but you cannot simply overflow

to system RAM without taking a catastrophic performance hit. The

bandwidth between CPU RAM and GPU VRAM is orders of magnitude slower

than VRAM bandwidth. When you hear about models being "quantized

to 4-bit," this is why 4-bit quantization halves the memory

footprint roughly, which is the difference between fitting on one

GPU and not fitting on one GPU.

Within the GPU itself, there is a memory hierarchy that determines

how fast computation runs. The KV cache, the cached attention

computation for the tokens already processed in a conversation

lives in VRAM and grows with sequence length. Managing KV cache

is one of the most consequential performance decisions in LLM serving,

and most teams don't think about it at all until they start hitting

out-of-memory errors on long contexts.

# KV cache management: what happens without it

# Each new token regenerates attention for the entire context

# Cost is O(n²) in sequence length



# What vLLM and similar systems do differently:

# PagedAttention manages KV cache in fixed-size blocks

# like virtual memory paging in an OS



# This allows:

# 1. Sharing KV cache between requests with the same prefix

# 2. Better memory utilisation (no internal fragmentation)

# 3. Handling variable-length sequences without pre-allocating

#    worst-case memory



from vllm import LLM, SamplingParams



# vLLM handles KV cache management automatically

# This is not a minor optimisation — it's 2-4x throughput improvement

# on typical workloads versus naive implementations



llm = LLM(

    model="meta-llama/Llama-3-8b-instruct",

    gpu_memory_utilization=0.90,    # Leave 10% headroom

    max_model_len=8192,             # Maximum sequence length

    enable_prefix_caching=True,     # Cache common prefixes (system prompts)

    tensor_parallel_size=1,         # Number of GPUs for this model

)



sampling_params = SamplingParams(

    temperature=0.7,

    max_tokens=512,

)



# Prefix caching means your system prompt is computed once

# and cached for all subsequent requests — significant for

# long system prompts used with every inference

outputs = llm.generate(prompts, sampling_params)

Most teams serving LLMs in production are not using PagedAttention.

They're using naive inference implementations that waste fifty to

seventy percent of their GPU memory to fragmentation and redundant

computation. The cost difference is not marginal.

The scaling question everyone asks wrong

When a team's AI infrastructure starts struggling under load, the

first question is almost always: "should we add more GPUs?"

This is the wrong question asked at the wrong time, for the same

reason that "should we add more database servers" was the wrong

first question when a database was struggling in 2008. The right

question is: "why are we using our current GPUs so inefficiently?"

GPU utilisation that is below sixty percent is almost always a

batching problem. Requests are not being grouped efficiently before

hitting the GPU. You can add more GPUs and halve your utilisation

number, which means you now have twice the infrastructure running at

thirty percent capacity instead of one set running at sixty. You've

doubled your cost and solved nothing.

GPU utilisation that is high but latency is still bad is almost

always a model sizing problem. The model is too large for the request

volume being served. A smaller quantized model, or a different

architecture, may serve your request latency requirements at a

fraction of the compute cost.

# Measuring what actually matters before deciding to scale



import time

import psutil

from prometheus_client import Histogram, Gauge, Counter



# These metrics tell you where the problem actually is



GPU_UTILISATION = Gauge(

    'gpu_utilisation_percent',

    'GPU compute utilisation',

    ['device_id']

)



GPU_MEMORY_USED = Gauge(

    'gpu_memory_used_bytes',

    'GPU VRAM in use',

    ['device_id']

)



BATCH_SIZE = Histogram(

    'inference_batch_size',

    'Number of requests processed per batch',

    buckets=[1, 2, 4, 8, 16, 32, 64]

)



TOKENS_PER_SECOND = Histogram(

    'inference_tokens_per_second',

    'Throughput of inference in tokens per second',

    buckets=[10, 25, 50, 100, 200, 400, 800]

)



TIME_TO_FIRST_TOKEN = Histogram(

    'inference_ttft_seconds',

    'Time from request to first token generated',

    buckets=[.05, .1, .25, .5, 1, 2, 5]

)



REQUEST_QUEUE_DEPTH = Gauge(

    'inference_queue_depth',

    'Number of requests waiting for GPU'

)





class InstrumentedInferenceServer:

    async def infer(self, prompts: list[str]) -> list[str]:

        BATCH_SIZE.observe(len(prompts))

        REQUEST_QUEUE_DEPTH.set(self.queue.qsize())



        start = time.perf_counter()

        results = await self._run_inference(prompts)

        duration = time.perf_counter() - start



        total_tokens = sum(len(r.split()) for r in results)

        TOKENS_PER_SECOND.observe(total_tokens / duration)



        return results

When you can see batch sizes, queue depth, tokens per second, and

time-to-first-token alongside GPU utilisation and VRAM usage, the

question of "do we need more GPUs" almost answers itself. Usually

the answer is "no, we need to batch better" or "no, we need to use

a smaller model" and scaling turns out to be unnecessary.

The cold start problem nobody planned for

Databases take seconds to start. GPU inference servers take minutes.

A database that restarts unexpectedly is back within thirty seconds

in most cases. An LLM inference server that restarts needs to load

model weights from storage into VRAM before it can serve any requests.

A 70B parameter model stored in 4-bit quantization is roughly 35GB.

Loading 35GB from network storage into VRAM, at typical cloud storage

bandwidth, takes several minutes under good conditions.

This changes incident dynamics entirely. A database blip is a brief

interruption. A GPU server blip is a several-minute outage for every

affected instance. Autoscaling, which works well for stateless

application servers and adequately for databases, works badly for

GPU inference because new instances take so long to become ready.

The teams that have worked this out run warm pools ,GPU instances

with models already loaded, sitting idle, waiting for traffic that

hasn't arrived yet. This feels wasteful. It's the only way to handle

traffic spikes without minutes-long latency blowouts.

# Kubernetes deployment with warm pool strategy

# Minimum replicas keep instances warm even at low traffic

# This costs money. The alternative is cold start latency.



apiVersion: apps/v1

kind: Deployment

metadata:

  name: llm-inference

spec:

  replicas: 3  # Never scale below this. These are your warm pool.

  template:

    spec:

      containers:

      - name: inference-server

        image: myteam/inference:latest

        resources:

          limits:

            nvidia.com/gpu: 1

          requests:

            nvidia.com/gpu: 1

        readinessProbe:

          httpGet:

            path: /health

            port: 8080

          # Model loading takes time. This probe must not pass

          # until the model is fully loaded in VRAM.

          initialDelaySeconds: 180   # 3 minutes minimum

          periodSeconds: 10

          failureThreshold: 30       # 5 more minutes of retries

        lifecycle:

          preStop:

            exec:

              # Drain in-flight requests before shutdown

              command: ["/bin/sh", "-c", "sleep 30"]



---

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

  name: llm-inference-hpa

spec:

  scaleTargetRef:

    apiVersion: apps/v1

    kind: Deployment

    name: llm-inference

  minReplicas: 3    # Warm pool floor

  maxReplicas: 10

  metrics:

  - type: External

    external:

      metric:

        name: inference_queue_depth

      target:

        type: AverageValue

        averageValue: "5"  # Scale when queue exceeds 5 requests per replica

  behavior:

    scaleUp:

      stabilizationWindowSeconds: 60

      policies:

      - type: Pods

        value: 2

        periodSeconds: 120   # Add 2 pods max every 2 minutes

                             # Fast enough to respond, slow enough

                             # to not over-provision during spikes

    scaleDown:

      stabilizationWindowSeconds: 600  # Wait 10 minutes before scaling down

                                       # Cold start cost makes yo-yo scaling

                                       # extremely expensive

The scaleDown stabilization window is long deliberately. The cold start

cost is so high that scaling down and back up in response to a brief

traffic dip is more expensive than just keeping the instances running.

This is counterintuitive if you're coming from stateless web services.

It's the operational reality of GPU infrastructure.

The cost model is upside down

Database costs scale with data volume and query complexity. You pay

more as your data grows and your queries get more complex.

GPU costs scale with time. You pay for every second a GPU exists,

whether it's serving requests or not. An idle GPU costs the same as

a busy GPU.

This inverts the normal infrastructure economics. With stateless

application servers, idle capacity is cheap, you can scale to zero

when traffic drops and pay nothing. With GPU inference, scaling to

zero means cold starts when traffic returns. The minimum viable

capacity for a production inference service is not zero, it's

whatever your warm pool needs to be, which is determined by your

acceptable cold start latency and your traffic spike patterns.

The teams that have made peace with this have stopped thinking about

GPU cost as a variable cost that tracks usage and started thinking

about it as a fixed cost that buys capacity. The question is not

"how do we pay less for GPU when traffic is low?" The question is

"what is the right amount of always-on capacity, and how do we make

sure we use it efficiently?"

Efficient use means high batch fill rates, high token throughput per

GPU-hour, low idle time. The metrics above are the inputs to this

calculation. Without them you're guessing about whether your

infrastructure is sized correctly.

The pattern that's emerging

The teams operating GPU infrastructure well in 2026 look, in their

operational discipline, a lot like the teams that operated databases

well in 2012 after enough people had been burned that the patterns

were starting to solidify.

They treat GPU utilisation as a lagging indicator and token throughput

as the leading one. They instrument everything: batch sizes, queue

depth, time-to-first-token, VRAM usage, KV cache hit rates. They

size their warm pools based on measured traffic patterns rather

than intuition. They run the smallest model that meets their quality

bar, not the largest model they can afford, because smaller models

batched efficiently outperform larger models batched poorly on

almost every practical metric.

They've also accepted something that takes a while to accept: that

the right abstraction for GPU infrastructure is not "fast compute"

but "throughput capacity." The question is not "how fast can this

machine process one request?" GPUs are fast at that regardless.

The question is "how many requests per dollar can this infrastructure

handle at acceptable latency?" That question requires different

metrics, different architecture, and a different mental model than

the one most teams bring from their experience with CPU infrastructure.

The database analogy runs deeper than it looks. In 2004, the teams

that treated the database as a black box, put data in, get data

out, add more RAM when it's slow, eventually hit walls that their

architecture couldn't get past. The teams that understood what was

happening inside the database query plans, index usage, lock

contention, buffer pool behaviour built things that scaled.

The GPU is not a black box. It has a memory hierarchy, a batching

model, a cost structure, and performance characteristics that reward

understanding and punish ignorance in the same way the database did.

The patterns are forming. The teams learning them now will have the

same advantage in five years that database-literate engineers had in 2015.

The mistakes are happening right now, at scale, expensively.

Most of them are the same mistakes. Most of them are avoidable.