Jetson Nano: Ollama & Optimal Quantization

I am delighted to announce that a user reported dysfunction so that I could go down the rabbit hole of fixing it. Messing around locally is one thing, but building a tolerable app for end users has other considerations when using a ‘local’ AI. This led to interesting findings about the limitations of hardware and gaining a high-level understanding of quantization and it's importance. I also explain how to creatively get around limitations. It's called flippy card, and it's an app that helps you study via custom uploaded content. Wanna see it? It's here:

After finally getting everything to work, the request took 13 minutes and 20 seconds using the Q8_0 version. This is where I learned about quantization. I then tried another variant, Q4_0 to improve the results.

Benchmarks

Here is the benchmark breakdown of Q8_0 vs Q4_K-M quantization:

Since the bottleneck was "model doesn't fit in available GPU memory," we tested a smaller quantization of the exact same model (llama3.2:1b), rather than switching to a different, weaker model family.

Q8_0 (original) Q4_K_M (new)

Model file size 1.5 GB 808 MB

GPU layers loaded 3–9 of 17 17 of 17 (100%)

Generation speed ~1.2–1.35 tok/s ~30.7 tok/s

Real 971-token test 13m 20s ~35–45s (estimated at this rate)

That's roughly a 25x speedup, because the entire model now fits on the GPU instead of mostly running on the slow CPU path. Amazing! But what's the catch?

Issues

Q4_K_M is fast but produced malformed output sometimes.

Before switching, ran 6 back-to-back test generations with Q4_K_M to check reliability, since lower-precision quantization can be less consistent.

Here's what happened:

• 2 of 6: perfectly valid JSON, correct structure


• 1 of 6: valid JSON, but used a slightly different field name than expected


• 3 of 6: malformed JSON (e.g., a mismatched bracket) that would have crashed the app's parser outright

That's roughly a 50-65% failure rate per attempt. I cannot knowingly ship that, even with a massive speed improvement.

So how do we handle the error-prone behavior of Q4?

Rather than giving up on the faster model, automatic retry logic was added to the app itself. If the model's response is malformed, the app now silently tries again up to 3 times before showing an error. Because each Q4_K_M attempt only takes about 30-45s, even a worst-case 3 attempts is still far faster than a single guaranteed-slow Q8_0 request, while pushing the effective success rate up to roughly 85-95%.

This handles the potential parsing errors gracefully. Since Q4 is 25x times faster, the client won’t really feel it.

Side note: I noticed my Jetson had the “super” abilities after a month of setting it up. Don’t do what I did. Check for super abilities first. It’s a free download. XD