LoRA: Fine-Tune a Giant Model by Training 1% of It

Fine-tuning a large model used to mean one painful thing: update every weight in it, keep a full copy per task, and pay for the GPUs to do it. A 7-billion-parameter model has 7 billion knobs. The Adam optimizer keeps two extra numbers per knob, so you're suddenly holding roughly three times the model in memory. Then, for every new task you tune, you store another 13 GB checkpoint. It works, but it's absurdly out of proportion to how much the model actually needs to change to, say, adopt a new tone or learn your domain's vocabulary.

LoRA — Low-Rank Adaptation — is the trick that makes that whole problem go away. And it's built on one clean observation.

The update is low-rank

When you fine-tune, what you really produce is a difference: the new weights minus the old weights, call it ΔW. The insight behind LoRA is that this difference has low "intrinsic rank." The useful change lives in a tiny subspace, not the full d×d space of the matrix. A big matrix of numbers can often be reconstructed almost perfectly from the product of two skinny matrices.

So instead of learning ΔW directly, LoRA learns two small factors whose product is ΔW:

ΔW ≈ B · A       (A is r×d,  B is d×r,  with r ≪ d)

Then it freezes the original weight W entirely and adds the detour on top. The effective weight the model uses becomes:

W + (α / r) · B · A

In the forward pass, an input x flows through both paths — Wx (the original behavior) plus B(Ax) (the learned correction). W never changes. The model starts out identical to the pretrained one and only drifts as you train A and B.

Why this is such a big win

Count the parameters. A d×d matrix has d² weights. The two adapter matrices together have r·d + d·r = 2·r·d weights. At d = 4096 and r = 8, that's about 65,000 trainable weights instead of 16.8 million on that layer — a 256× reduction. Across a whole model you routinely end up training well under 1% of the parameters.

Because only A and B get gradients, the optimizer only tracks state for those. Memory collapses along with the parameter count, and suddenly you can fine-tune a 7B–70B model on a single consumer GPU instead of a cluster.

Two knobs control it. Rank r sets the capacity of the detour — bigger r expresses a richer change at more cost. Alpha (α) is a scaling factor applied as (α/r)·BA, letting you tune how hard the adapter pushes independently of r, so bumping the rank doesn't force you to re-tune your learning rate. A common recipe is α ≈ 2r.

One detail people miss: A starts as small random values, B starts at exactly zero. That makes B·A zero at step 0, so training begins from the pristine base model and only departs as it learns. No random jolt to the weights on the first step.

Swap it, or bake it in

Here's the part that makes LoRA feel like magic in production. The trained adapter is a standalone file — often a few megabytes, not gigabytes — that references the shared frozen base. So you train dozens of them: one per task, tone, customer, or language. The giant base is stored once, and you hot-swap a few megabytes to switch behaviors at request time. That's how one hosted model serves many specialized variants cheaply, and how communities share thousands of style adapters for the same base.

If you'd rather have zero runtime cost, you merge instead. Compute W + (α/r)·BA once, overwrite W, and ship a model that looks completely ordinary — no LoRA branch, no extra matrix multiply, no added latency. You trade swappability for raw speed.

QLoRA takes it further: load the frozen base in 4-bit precision, train the adapters in higher precision on top. The base costs a quarter of the memory while the small trainable part stays accurate — enough to fine-tune a 65B model on a single 48GB card. It's the standard consumer-hardware recipe today.

The catch

LoRA assumes the change you need is low-rank. That holds for most adaptations. But if you're teaching genuinely new, complex behavior far from anything the base saw in pretraining, a tiny r simply can't express it — the loss plateaus above zero no matter how long you train. The fixes are to raise r, adapt more modules, or fall back to full fine-tuning. LoRA is a superb default, not a universal replacement.

In practice you never hand-write A and B. Hugging Face's peft does it all through one LoraConfig — set r, lora_alpha, and target_modules, wrap your model with get_peft_model, and everything else freezes automatically.

I built an interactive version where you can drag the rank, watch the trainable-parameter count collapse, and actually train the A and B matrices by gradient descent — including the case where a low rank can't fit a full-rank target. Play with it here: https://dev48v.infy.uk/ai/days/day24-lora.html