Running Gemma 4 26B on a 13-Year-Old Xeon: Practical AI Performance Without GPUs

Originally published on tamiz.pro.

Introduction

Large language models like Gemma 4 26B typically require powerful GPUs with high VRAM. This tutorial demonstrates how to run the model on a 13-year-old Xeon processor (e.g., Intel Xeon E5 v2 series) using CPU-only optimization techniques like model quantization and memory-efficient execution.

Prerequisites

• Xeon-based server with at least 64GB RAM


• Linux OS (Ubuntu/CentOS recommended)


• Python 3.10+


• 
  git, cmake, and gcc installed


• At least 200GB free disk space

Step 1: Prepare the Environment

Start by installing core dependencies:

sudo apt-get update

sudo apt-get install -y python3-pip build-essential

pip install torch==2.1.0 transformers optimum

Verify PyTorch's CPU support with:

import torch

print(torch.__version__, torch.cuda.is_available())  # Should return False

Step 2: Download and Convert the Model

Use Hugging Face's from_pretrained with quantization:

from transformers import AutoModelForCausalLM, AutoTokenizer



model_id = "google/gemma-4-26b"

tokenizer = AutoTokenizer.from_pretrained(model_id)



# Load with 4-bit quantization

model = AutoModelForCausalLM.from_pretrained(

    model_id,

    load_in_4bit=True,

    torch_dtype=torch.float16

)

This reduces RAM usage from 120GB (float16) to ~40GB through quantization.

Step 3: Optimize Memory Usage

Add CPU-specific optimizations:

from torch._dynamo import optimize_for_cpu



# Enable optimized CPU execution

model = optimize_for_cpu(model)

model.tie_weights()



# Configure attention computation

import torch.nn as nn

nn.Linear(model.config.hidden_size, model.config.hidden_size).to(memory_format=torch.channels_last)

Step 4: Run Inference

Execute with batch size 1 and CPU-optimized pipeline:

input_text = "Explain quantum computing in simple terms"

inputs = tokenizer(input_text, return_tensors="pt")



# Use CPU for inference

with torch.no_grad():

    outputs = model.generate(**inputs, max_new_tokens=100)

    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance Expectations

Optimization Tips

1. Use load_in_8bit instead of load_in_4bit if RAM is constrained


2. Disable gradient computation globally


3. Use --cpu-inference flag in any training scripts


4. Enable Intel MKL optimizations:
   

export MKL_THREADING_LAYER=GNU

export MKL_SERVICE_FORCE_INTEL=1

Conclusion

While modern GPUs provide better throughput (100-300 tokens/sec), this CPU-only approach enables AI inference on legacy hardware at ~15% of GPU costs. Ideal for edge deployments or proof-of-concept work. Consider upgrading to Xeon Scalable (2nd Gen) for production workloads requiring higher throughput.