What a GPU Actually Is (and Why ML Stole It)

Introduction

You've written model.to('cuda') a hundred times. You've celebrated when training loss went down. You've cursed when CUDA out of memory killed your run at 3am.

But here's a question: do you actually know what happened inside that GPU?

Not vaguely. Not "it's parallel" as a hand-wave. Do you know why a 4096×4096 matrix multiply finishes in 12 milliseconds on a GPU but takes 800 milliseconds on a CPU same math, same numbers, same code structure?

If not, you're driving a Formula 1 car using only first gear. And that's exactly what most ML engineers do.

This article is the foundation. Everything else in GPU optimization mixed precision, FlashAttention, quantization, vLLM is just a clever trick that exploits something about how GPUs physically work. If you understand the machine, the tricks become obvious. If you don't, they're just magic spells you copy from blog posts.

By the end of this, you'll know:

• Why GPUs have 10,000+ cores but can't run a simple if/else efficiently


• The memory hierarchy that makes or breaks every optimization you'll ever apply


• What's actually inside a Streaming Multiprocessor (the thing nvidia-smi reports on)


• Why your $15/hr Colab T4 shares more DNA with a $40K H100 than you think

No CUDA programming required. No C++ required. Just Python basics and curiosity.

Section 1: The CPU vs GPU Two Fundamentally Different Philosophies

Let's start with an analogy that actually works.

A CPU is a brilliant professor. Give them any problem parse a sentence, evaluate a complex conditional, jump between 14 different tasks they'll solve it fast. They have a massive personal library (cache), they can predict what you'll ask next (branch prediction), and they handle interruptions gracefully. But there's only 8-16 of them in your machine.

A GPU is a stadium full of factory workers. Each one can only do simple arithmetic add these two numbers, multiply those two. They can't improvise. They can't handle surprises. But there are ten thousand of them, and if you give all of them the same instruction on different data, they finish in the time it takes the professor to sharpen their pencil.

This isn't a metaphor I'm stretching. It's literally how the silicon is designed:

Why Matrix Multiplication Is the Perfect GPU Workload

Here's the thing that made NVIDIA a trillion-dollar company:

Neural networks are, computationally, almost entirely matrix multiplication. The forward pass? Matrix multiply. Attention? Matrix multiply. The backward pass? More matrix multiplies. When you train GPT-4 or run inference on Llama, roughly 95%+ of the compute is just C = A × B on huge matrices.

Now think about what a matrix multiply actually requires. To compute one element C[i][j], you multiply row i of A by column j of B and sum the results. Here's the key insight: every element of C is independent. C[0][0] doesn't need C[0][1] to be done first. They're all separate dot products.

A 4096×4096 output matrix has 16.7 million elements. Each one is an independent dot product. That's 16.7 million tasks that share the same instruction ("do a dot product") but operate on different data.

This is exactly what GPUs are built for. Same instruction. Different data. Millions of times.

The professor could do each dot product perfectly, one by one. But the factory workers, doing all 16.7 million simultaneously? They'll finish before the professor gets through 1% of them.

The Deeper Design Principle: Throughput vs Latency

This goes beyond "more cores." CPUs and GPUs represent two fundamentally different bets on how to spend transistors:

CPUs bet on latency. Make each individual task finish as fast as possible. Devote transistors to:

• Branch prediction (guess what code runs next, start executing before you know)


• Out-of-order execution (rearrange instructions for speed)


• Huge L1/L2/L3 caches (keep data close so you never wait)

GPUs bet on throughput. Accept that any one task might be slow, but process thousands simultaneously. Devote transistors to:

• More arithmetic units (the actual computation lanes)


• More registers (so thousands of threads can stay "live" simultaneously)


• Warp schedulers that switch between thread groups at zero cost when one stalls

Here's the counterintuitive consequence: a single GPU thread is slower than a single CPU thread. If you run one multiplication on a GPU, the CPU wins. The GPU only wins when you give it enough parallel work to keep all its cores busy.

This is why batch size matters. This is why you can't train with batch_size=1 efficiently. You need to feed the machine enough parallel work, or you're paying for 10,000 workers and only using 12 of them.

That batch_size insight? It'll come back in every article in this series. But first, you need to understand where data lives inside this machine because that's where the real bottleneck hides.

Section 2: The Memory Hierarchy Where Every Bottleneck Actually Lives

Here's a dirty secret about GPU optimization: almost none of it is about computation. Your GPU can multiply numbers so fast that most of the time it's sitting idle, waiting for data to arrive. The bottleneck isn't doing the math it's getting the numbers to the math units in time.

GPUs are almost never compute-bound. They're memory-bound.

Every optimization you'll encounter in this series FlashAttention, quantization, KV caching, kernel fusion is fundamentally an attack on memory movement, not computation. Once you understand where data lives and how fast it moves between layers, every optimization in ML becomes obvious. Let's look at the layout.

Two quick definitions before we dive in we'll explore both in depth in later sections, but you'll see them in the table below:

• 
  SM (Streaming Multiprocessor) the mini-processor inside a GPU that actually runs your threads. A GPU has 40-132 of them. We'll explore these in Section 3.


• 
  HBM (High Bandwidth Memory) the main GPU memory (what nvidia-smi shows as "GPU Memory"). It's the big pool of VRAM (16-192 GB) where your model weights and tensors live.

If you don't know what tensors are? A tensor is just a multi-dimensional array of numbers. That's it. The word sounds intimidating but the concept is simple.

The Five Layers

Read that speed column again. From SRAM to HBM is a 10-60x bandwidth drop, and from HBM to CPU RAM via PCIe is another 60-100x drop. That's not a gradual slope it's a cliff, then another cliff. These cliffs are where all the time gets lost.

The Cliff That Explains .to('cuda')

Remember that fifth layer CPU RAM, connected via PCIe? When you write tensor.to('cuda'), you're copying data across that outermost, slowest link. PCIe 4.0 x16 gives you ~32 GB/s; HBM bandwidth on the A100 is ~2,000 GB/s. That's a 60x speed difference. Moving a 7B parameter model (14 GB in FP16) from CPU to GPU takes nearly half a second over PCIe, but once it's on the GPU, processing it is almost instant by comparison.

This single bottleneck explains a surprising number of best practices: dataloaders prefetch batches to GPU memory ahead of time, you never call .to('cuda') inside a training loop, multi-GPU setups use NVLink (900 GB/s on H100) instead of PCIe to communicate between GPUs, and model loading time dominates startup rather than inference.

The Map of Every Optimization in This Series

Here's the beautiful part. Every optimization technique we'll cover is just an attack on a specific layer of this hierarchy:

Every single one targets memory movement. Not compute. We'll build each of these in later articles but now you'll understand why they work.

Three Things to Take Away

The sentence to memorize: your GPU does math 300x faster than it can read memory every optimization in ML is fundamentally about reducing memory movement, not speeding up arithmetic.

If you remember nothing else from this section:

1. The memory hierarchy has cliffs, not slopes. SRAM → HBM is 10-60x slower. HBM → PCIe is another 60x. These gaps are where time is lost.




2. Most ML operations are memory-bound, not compute-bound. The GPU's math units sit idle most of the time, starved for data. Arithmetic intensity (ops per byte) determines everything.




3. Every major optimization is a memory trick. FlashAttention = fewer HBM reads. Quantization = smaller data. Kernel fusion = fewer round-trips. Once you see this, the whole field makes sense.

Now that you know the terrain the machine and its memory you need to understand the organization of the workers themselves. How does a GPU take 10,000 threads and coordinate them without chaos? That's the Streaming Multiprocessor, and it's more elegant than you'd expect.

Section 3: Inside the GPU SMs, Warps, Cores, and How Work Gets Organized

You now know what a GPU is (a throughput machine) and where data lives (a memory hierarchy with cliffs). The missing piece: how does a GPU actually organize 10,000 threads without descending into chaos?

The answer is a surprisingly clean hierarchy. Let's zoom in from the top.

The Streaming Multiprocessor: The GPU's Fundamental Building Block

Remember the factory from Section 1 thousands of workers doing simple arithmetic? Here's the thing the analogy skipped: those workers aren't all standing in one giant room. They're organized into departments.

A GPU isn't one giant processor. It's a collection of 40 to 132 smaller, self-contained processors called Streaming Multiprocessors (SMs). Each SM is a complete mini-engine it has its own compute units (the workers), its own fast scratchpad memory (the Shared Memory / SRAM from Section 2), its own private registers, and its own scheduler that decides what to run next. An SM doesn't need to ask the rest of the GPU for permission to do anything. It receives a chunk of work, and it grinds through it independently.

Think of the GPU as an office building. Each SM is a self-sufficient department floor it has its own desks, its own whiteboard, its own team lead. The building has shared plumbing (HBM and L2 cache, which all floors access), but each floor operates independently. When a big job comes in, the building's front desk (the GPU's global scheduler) splits it into chunks and hands one to each floor. The floors don't coordinate with each other they just crunch their assigned chunk and report back.

Why does the number of SMs matter? Because it's the GPU's parallelism ceiling. If you have 108 SMs (A100) and your workload only generates 40 chunks of work, then 68 SMs sit completely idle. More SMs = more chunks running simultaneously = more throughput. This is also why nvidia-smi reports SM-level utilization it's telling you how many floors in the building are actually occupied.

But saying "an SM has compute units" is vague. What kind of compute units? It turns out there are two very different types inside each SM, and understanding the difference is essential.

CUDA Cores and Tensor Cores: Two Kinds of Workers

Inside each SM sit two types of compute units, and understanding the difference matters for everything that comes later.

CUDA Cores are general-purpose arithmetic units. Each one handles a single floating-point multiply-add per clock cycle one number in, one number out. They're versatile: any FP32 or FP64 math goes through them. An A100 SM has 64 CUDA cores, giving it 6,912 total across all 108 SMs. When people say "the GPU has thousands of cores," these are what they mean.

Tensor Cores are specialists built for one job: matrix multiplication. Instead of processing one multiply-add per cycle like a CUDA core, a Tensor Core crunches an entire small matrix operation in a single shot many multiply-adds simultaneously. Neural networks spend almost all their time doing matrix multiplication, so NVIDIA built dedicated silicon to do it faster. How much faster? The A100's Tensor Core throughput is 312 TFLOPS. Its CUDA-core-only throughput is about 20 TFLOPS. That's roughly 16x slower for the same operation.

This distinction will become critical in Article 5 (mixed precision) because Tensor Cores only operate on certain data types. When you switch from FP32 to FP16 or BF16, you're not just halving the memory you're unlocking Tensor Cores, which is where the real speedup comes from. On a GPU without Tensor Cores (older than 2017), FP16 gives you a marginal improvement. On a GPU with them, it's a 8-16x leap.

Now that you know what SMs, CUDA Cores, and Tensor Cores actually are, here's how they stack up across the GPUs you'll see throughout this series:

Notice the pattern: Tensor Core TFLOPS dwarfs CUDA Core TFLOPS by 8-15x. When someone quotes a GPU's "performance," they're almost always quoting the Tensor Core number. And when your code isn't using Tensor Cores, you're leaving 90%+ of the silicon on the table.

Warps: The Real Unit of Execution

Here's something most ML engineers never learn, and it's arguably the most important concept for understanding GPU behavior: the GPU doesn't execute individual threads. It executes warps groups of exactly 32 threads that move in perfect lockstep.

When an SM runs your code, it doesn't look at threads one by one. It grabs 32 threads, gives them all the same instruction ("multiply register A by register B"), and all 32 execute that instruction simultaneously each on its own data. Next clock cycle, same thing: one instruction, 32 threads. This model is called SIMT (Single Instruction, Multiple Threads), and the group of 32 is a warp.

Why does this matter? Because it has a direct consequence for branching. If threads within a warp hit an if/else and disagree on which way to go, the GPU can't split them the warp is physically locked together, so both paths end up running and performance suffers. This is called warp divergence, and it's important enough that Section 4 is dedicated entirely to understanding how and why it happens.

The number 32 isn't arbitrary it's baked into the hardware. You'll see it everywhere in GPU programming: thread blocks should be multiples of 32, memory access patterns align to warps, Tensor Core operations are warp-wide. When someone says "think in warps, not threads," this is what they mean.

Threads, Blocks, and Grids: How Your Code Maps to Hardware

So you have SMs, and each SM runs warps of 32 threads. But how does your PyTorch code actually turn into threads on SMs? The mapping works through three levels:

A thread is the smallest unit one execution lane doing one piece of work (like computing one element of an output matrix). Threads are grouped into blocks of 32 to 1,024 threads. A block is important because it lives on exactly one SM, and all threads in a block share that SM's Shared Memory and can synchronize with each other. Blocks are grouped into a grid, which is the entire launch of work for one GPU operation.

The beauty of this model is separation of concerns. You, the programmer (or PyTorch, on your behalf), define how many blocks and threads you need based on the problem size. The GPU's hardware scheduler distributes those blocks across available SMs you don't choose which SM runs which block. If you have 256 blocks and 108 SMs, every SM has work. If you have 40 blocks and 108 SMs, most SMs sit idle. This is why batch size matters so directly to GPU utilization a larger batch means more blocks, more SMs occupied, less hardware wasted.

When nvidia-smi shows "GPU-Util: 87%", it's roughly measuring whether SMs had any blocks to run. But it doesn't tell you how efficiently those blocks used their SM that requires deeper profiling, which we'll cover in later articles.

Putting It All Together: The Full Picture

Let's zoom out and see how everything nests:

And it gets worse with more branches. A 4-way switch statement? Up to 4 serial passes through the same warp. A nested if-else tree? Each level multiplies the potential serialization. The worst case is when every thread in a warp takes a different path you get zero parallelism from 32 cores.

The practical consequence: any operation that makes threads within a warp do fundamentally different things is a poor fit for GPUs. Data-dependent branching, variable-length sequence processing, tree traversals all of these cause warps to diverge. This is one reason why Transformer attention (mostly uniform matrix math across tokens) displaced RNNs (sequential, branchy) in modern ML Transformers are structurally friendlier to GPU hardware.

Sequential Dependencies: When Step N Needs Step N-1

GPUs win by doing millions of things at once. But what if each step depends on the result of the previous one?

Think about a simple loop: x = f(x) repeated 1,000 times. You can't start iteration 500 until iteration 499 finishes, because you need its output as input. This is inherently sequential there are no independent tasks to parallelize. Ten thousand cores and one core will finish at the same speed, because only one core has any work to do at a time.

This pattern shows up in surprisingly important places:

Autoregressive text generation is the big one. When an LLM generates text, each new token depends on all previous tokens. Training processes all tokens in parallel (because you know the target sequence ahead of time), but generation is one token at a time, each waiting for the last. This single constraint is why inference is so much slower than training, and it's why half the optimizations in this series exist KV caching, speculative decoding, continuous batching all working around the sequential bottleneck of autoregressive decoding.

Recursive algorithms tree search, depth-first graph traversal, dynamic programming where each cell depends on previous cells hit the same wall. The dependency chain prevents parallelism. GPUs can parallelize across independent branches of a tree, but they can't speed up a single path through it.

General-purpose control flow parsing, compilation, OS scheduling, database query planning these are decision-heavy, branchy, sequential tasks where the CPU's branch predictor and out-of-order execution absolutely dominate. No amount of GPU cores helps when the work is "decide what to do next based on what just happened."

The pattern is clear: if your computation is a long chain where each link depends on the previous one, a GPU can't help. It needs thousands of independent tasks to stay busy.

Small Workloads: The Launch Tax

Even when work is perfectly parallel, there's a minimum viable size below which GPUs actually lose to CPUs.

Launching a GPU kernel isn't free. The CPU has to package the kernel parameters, send them across PCIe to the GPU, wait for the GPU's scheduler to pick up the work, and collect results. This round-trip called kernel launch overhead takes roughly 5-15 microseconds even for the simplest operation. If the actual computation takes 2 microseconds, you've spent more time on the overhead than on the work itself.

This is why operations on tiny tensors are faster on the CPU. Matrix multiply on [4096, 4096]? GPU wins by 100x. Matrix multiply on [8, 8]? CPU wins because the GPU hasn't even finished its kernel launch by the time the CPU is done. The GPU is a freight train: incredible throughput once it's moving, but it takes time to start. For small packages, a bicycle gets there first.

Why This Is Fine: Know Your Splits

Here's the key insight that makes all of this practical instead of academic: the CPU and GPU aren't competitors they're partners. Every real ML pipeline uses both, and the job of a good engineer is knowing which operation belongs on which hardware.

Here's how a typical training loop actually splits:

The CPU handles the branchy, sequential, small-batch work. The GPU handles the matrix-heavy, parallel, large-batch work. They work in a pipeline: the CPU prepares the next batch while the GPU trains on the current one. This is exactly what DataLoader(num_workers=4) does it spins up CPU workers to prepare data ahead of time so the GPU never waits.

When people say "my GPU utilization is low," the problem is almost always one of three things: the CPU isn't feeding data fast enough (data pipeline bottleneck), the batch size is too small (not enough parallel work to fill the SMs), or there's too much CPU↔GPU transfer (calling .to('cuda') in a loop). All three are mismatches between the workload and the hardware not a limitation of the GPU itself.

Three Things to Take Away

The sentence to memorize: GPUs don't have weaknesses they have boundaries, and respecting those boundaries is the first real optimization.

If you remember nothing else from this section:

1. Branching kills GPU performance. Warp divergence forces both paths to run serially. The more threads in a warp disagree on which branch to take, the more time is wasted. Design workloads so threads in a warp do the same thing.




2. Sequential dependencies can't be parallelized. If step N needs step N-1's result, 10,000 cores won't help. This is why autoregressive generation is slow and why training (parallel) is fundamentally more efficient than inference (sequential).




3. CPU and GPU are partners, not competitors. Branchy work on CPU, matrix work on GPU. The pipeline between them DataLoader, prefetching, minimizing transfers is where most real-world performance is won or lost.

You now have the complete mental model: the machine, its memory, its organization, and its limits. The only thing left is to see it with your own eyes. Let's fire up a GPU and look under the hood.

Let's Prove It: Four Demos on a T4

Everything above is theory. Let's prove each point with code you can run right now on a free Colab T4. Copy these cells into a notebook and run them in order.

Setup run this cell first:

import torch

import time



assert torch.cuda.is_available(), \

    "No GPU found. Enable it: Runtime > Change runtime type > T4 GPU"



def bench_gpu(fn, warmup=20, iters=100):

    """Benchmark a GPU operation with proper CUDA synchronization."""

    for _ in range(warmup):

        fn()

    torch.cuda.synchronize()

    start = time.perf_counter()

    for _ in range(iters):

        fn()

    torch.cuda.synchronize()

    return (time.perf_counter() - start) / iters * 1000  # ms



def bench_cpu(fn, warmup=5, iters=50):

    """Benchmark a CPU operation."""

    for _ in range(warmup):

        fn()

    start = time.perf_counter()

    for _ in range(iters):

        fn()

    return (time.perf_counter() - start) / iters * 1000  # ms



print(f"GPU: {torch.cuda.get_device_name()}")

print(f"CUDA: {torch.version.cuda}")  # software toolkit version, NOT the hardware cores

The conditional version is measurably slower typically 1.5-3x on a T4. The overhead comes from evaluating the condition, handling two code paths, and potential warp divergence when threads within a warp disagree on which branch to take. PyTorch's built-in CUDA kernels are well-optimized to minimize the damage, so the gap here is modest. In raw CUDA with heavy branching (nested if-else trees, switch statements), the penalty grows much steeper which is exactly why PyTorch's kernel engineers work so hard to keep branching out of the hot path.

Demo 2: Sequential vs Parallel The Autoregressive Tax

This is the dramatic one. We simulate autoregressive generation (each matmul depends on the previous output, so steps must run one at a time) vs batched computation (all rows processed at once). Same total FLOPs. Completely different performance.

hidden = 1024

seq_len = 128

W = torch.randn(hidden, hidden, device='cuda')



# Sequential: 128 matmuls, each feeding into the next

def sequential():

    x = torch.randn(1, hidden, device='cuda')

    for _ in range(seq_len):

        x = x @ W   # can't start until the previous x is ready

    return x



# Parallel: one batched matmul, same total FLOPs

def parallel():

    x = torch.randn(seq_len, hidden, device='cuda')

    return x @ W   # all 128 rows processed at once



seq_ms = bench_gpu(sequential, warmup=10, iters=50)

par_ms = bench_gpu(parallel, warmup=10, iters=50)



print(f"Sequential (128 dependent matmuls):  {seq_ms:.2f} ms")

print(f"Parallel   (1 batched matmul):       {par_ms:.2f} ms")

print(f"Speedup:                             {seq_ms/par_ms:.0f}x")

CPU wins at sizes 8 and 32 (sometimes even 128). GPU takes over somewhere around 128-512 and then dominates at 4096, it's 50-200x faster. The crossover exists because every GPU kernel launch has a fixed overhead of ~5-15 microseconds. When the actual compute takes less time than that overhead, the freight train loses to the bicycle.

One subtlety in the code: lambda a=a_cpu, b=b_cpu: a @ b captures the tensors at definition time using default arguments. Without a=a_cpu, Python's closures capture variables by reference every iteration of the loop would benchmark the last matrix size only. A common Python gotcha worth knowing.

Demo 4: The Transfer Tax What .to('cuda') Actually Costs

Moving data between CPU and GPU crosses PCIe the outermost, slowest cliff in the memory hierarchy from Section 2.

sizes_mb = [1, 10, 100, 500]



print(f"{'Size':>8}  {'CPU to GPU (ms)':>16}  {'Effective BW':>14}")

print("-" * 42)



for mb in sizes_mb:

    n_elem = mb * 1024 * 1024 // 4   # FP32 = 4 bytes per element

    x = torch.randn(n_elem)          # lives on CPU



    torch.cuda.synchronize()

    t0 = time.perf_counter()

    for _ in range(10):

        _ = x.cuda()                  # CPU to GPU transfer

        torch.cuda.synchronize()

    ms = (time.perf_counter() - t0) / 10 * 1000



    gbps = (mb / 1000) / (ms / 1000)  # GB/s

    print(f"{mb:>5} MB  {ms:>14.2f} ms  {gbps:>10.1f} GB/s")



print(f"\nCompare: HBM bandwidth is ~320 GB/s on T4, ~2,000 GB/s on A100.")

print(f"PCIe is 10-30x slower. This is the .to('cuda') tax.")

A 500MB tensor (roughly a 125M-parameter model in FP32) takes 100-150ms to cross PCIe. Once it's on the GPU, processing it takes microseconds by comparison. This single number explains why you load the model to GPU once at startup, why DataLoader uses pin_memory=True to speed up transfers, and why calling .to('cuda') inside a training loop is one of the most common beginner performance mistakes.

Three Things to Take Away

The sentence to memorize: GPUs don't have weaknesses they have boundaries, and respecting those boundaries is the first real optimization.

If you remember nothing else from this section:

1. Branching kills GPU performance. Warp divergence forces both paths to run serially. The more threads in a warp disagree on which branch to take, the more time is wasted. Design workloads so threads in a warp do the same thing.




2. Sequential dependencies can't be parallelized. If step N needs step N-1's result, 10,000 cores won't help. This is why autoregressive generation is slow and why training (parallel) is fundamentally more efficient than inference (sequential).




3. CPU and GPU are partners, not competitors. Branchy work on CPU, matrix work on GPU. The pipeline between them DataLoader, prefetching, minimizing transfers is where most real-world performance is won or lost.

You now have the complete mental model and the benchmarks to prove it.

What You Now Know (That Most ML Engineers Don't)

Let's take stock. In one article, you've built a mental model that most ML engineers never acquire even after years of writing model.to('cuda'):

You know a GPU isn't a faster CPU. It's a fundamentally different machine thousands of tiny cores that trade single-thread speed for massive parallel throughput. You know why matrix multiplication is the perfect workload for it (16.7 million independent dot products), and why batch size directly controls whether those cores stay busy or sit idle.

You know the memory hierarchy has cliffs, not slopes SRAM to HBM is a 10-60x drop, HBM to PCIe is another 60x. You know the GPU's math units are 312x faster than its memory pipe, which means almost every optimization in ML is a memory trick, not a compute trick. FlashAttention, quantization, kernel fusion you can now explain why each one works in one sentence.

You know what's inside an SM (CUDA Cores for general math, Tensor Cores for matrix ops at 16x the speed), how warps of 32 threads execute in lockstep, and how blocks map your code to SMs. You know what GPUs are terrible at branching, sequential dependencies, tiny workloads and why that's fine, because the CPU handles those jobs while the GPU handles the matrix math.

And you proved all of it with code. You measured the branch tax, the sequential-vs-parallel gap, the GPU-vs-CPU crossover point, and the PCIe transfer cost. Those aren't facts you read they're numbers you generated on real hardware. They'll stick.

You went from "it's parallel" as a hand-wave to understanding the actual machine. That's the foundation everything else in this series builds on.

What's Coming in Article 2: Your First Tensor on GPU

You understand the machine. Now it's time to use it.

Article 2 is where the rubber meets the road. You'll write your first properly benchmarked GPU code and immediately discover that most GPU benchmarks online are wrong (because they forget torch.cuda.synchronize(), something you already know matters from this article's demos).

This is Article 1 of a 12-part series on GPU optimization for ML engineers. Next up: Article 2 Your First Tensor on GPU (CPU vs GPU, Same Code)