Fine-tuning — Domain-Specializing Models with LoRA

[RAG]

Question → search DB → pass results to LLM → answer

→ Requires documents, search costs apply



[Fine-tuning]

Question → Fine-tuned LLM → answer

→ Model carries the knowledge itself — no retrieval needed

Full Fine-tuning:

  Updates all parameters (billions) of the model

  → High accuracy but requires GPU — expensive

  → 7B model needs minimum 16GB VRAM



LoRA (Low-Rank Adaptation):

  Trains only 0.1–1% of all parameters

  → Works on CPU, low cost

  → Accuracy close to full fine-tuning

Original model weight matrix W (unchanged)

    ↓

Add low-rank matrices A × B (only these are trained)

    ↓

At inference: compute W + α × (A × B)

pgvector-tutorial/

├── existing files

└── finetuning/

    ├── prepare_dataset.py    # ★ Dataset preparation

    ├── train_lora.py         # ★ LoRA fine-tuning

    ├── evaluate.py           # ★ Comparison with base model

    └── inference.py          # ★ Inference with trained model

pip install transformers datasets peft accelerate torch

pip freeze > requirements.txt

# finetuning/prepare_dataset.py

"""

Fine-tuning dataset preparation



Create question → answer pairs from pgvector documents.

"""

import json

import os



# ── Manually created Q&A pairs ────────────────────────────────

# Data quality is paramount for fine-tuning.

# In production, create 100–1000 high-quality pairs by hand.

TRAINING_DATA = [

    {

        "instruction": "What is the F1 score? Explain briefly.",

        "input": "",

        "output": "The F1 score is the harmonic mean of Precision (positive predictive value) "

                  "and Recall (sensitivity). "

                  "F1 = 2 × Precision × Recall ÷ (Precision + Recall). "

                  "It's especially useful when class imbalance is present.",

    },

    {

        "instruction": "Show me code for cross-validation with scikit-learn.",

        "input": "",

        "output": "from sklearn.model_selection import cross_val_score\n"

                  "from sklearn.ensemble import RandomForestClassifier\n\n"

                  "model = RandomForestClassifier()\n"

                  "scores = cross_val_score(model, X, y, cv=5, scoring='f1')\n"

                  "print(f'F1 Score: {scores.mean():.3f} ± {scores.std():.3f}')",

    },

    {

        "instruction": "What are the main methods for handling missing values in Pandas?",

        "input": "",

        "output": "There are three main approaches:\n"

                  "1. Drop: df.dropna() removes rows with missing values\n"

                  "2. Fill: df.fillna(value) replaces with a specified value (mean, median, etc.)\n"

                  "3. Interpolate: df.interpolate() fills using surrounding values\n"

                  "Numeric columns typically use mean imputation; categorical columns use mode.",

    },

    {

        "instruction": "Give me three ways to reduce EC2 costs on AWS.",

        "input": "",

        "output": "1. Use Spot Instances: up to 90% cheaper than On-Demand. "

                  "Best for interruptible batch workloads.\n"

                  "2. Reserved Instances: 1–3 year commitments for up to 75% discount.\n"

                  "3. Delete unused resources: regularly remove idle EBS volumes, EIPs, and snapshots.",

    },

    {

        "instruction": "What is a Kubernetes Pod?",

        "input": "",

        "output": "A Pod is the smallest deployable unit in Kubernetes.\n"

                  "It groups one or more containers that run on the same node, "

                  "sharing storage, networking, and execution specifications.\n"

                  "Pods are defined using YAML manifest files.",

    },

    {

        "instruction": "What is the difference between Precision and Recall?",

        "input": "",

        "output": "Precision (positive predictive value): of all predicted positives, "

                  "the fraction that are actually positive. TP / (TP + FP). "

                  "Prioritize when false positives are costly.\n"

                  "Recall (sensitivity): of all actual positives, "

                  "the fraction correctly detected. TP / (TP + FN). "

                  "Prioritize when false negatives are costly.",

    },

    {

        "instruction": "How do you select specific columns from a Pandas DataFrame?",

        "input": "",

        "output": "Several approaches are available:\n"

                  "Single column: df['column_name'] or df.column_name\n"

                  "Multiple columns: df[['col1', 'col2']]\n"

                  "Label-based: df.loc[condition, 'column']\n"

                  "Integer-based: df.iloc[row_index, col_index]",

    },

    {

        "instruction": "What is the difference between AWS S3 and EBS?",

        "input": "",

        "output": "S3 (Simple Storage Service): Object storage. "

                  "Manages files by key. Scalable and inexpensive. Ideal for static files and backups.\n"

                  "EBS (Elastic Block Store): Block storage. "

                  "Attached to EC2 instances like an HDD/SSD. "

                  "Used for databases and OS disks.",

    },

]





def save_dataset(data: list[dict], output_file: str):

    """Save dataset in JSONL format with 80/20 train/val split."""

    split_idx = int(len(data) * 0.8)

    train_data = data[:split_idx]

    val_data = data[split_idx:]



    train_file = output_file.replace(".jsonl", "_train.jsonl")

    val_file = output_file.replace(".jsonl", "_val.jsonl")



    with open(train_file, "w", encoding="utf-8") as f:

        for item in train_data:

            f.write(json.dumps(item, ensure_ascii=False) + "\n")



    with open(val_file, "w", encoding="utf-8") as f:

        for item in val_data:

            f.write(json.dumps(item, ensure_ascii=False) + "\n")



    print(f"Training data: {len(train_data)} samples → {train_file}")

    print(f"Validation data: {len(val_data)} samples → {val_file}")

    return train_file, val_file





def format_prompt(item: dict) -> str:

    """Convert to Alpaca format."""

    if item["input"]:

        return (

            f"### Instruction:\n{item['instruction']}\n\n"

            f"### Input:\n{item['input']}\n\n"

            f"### Response:\n{item['output']}"

        )

    return (

        f"### Instruction:\n{item['instruction']}\n\n"

        f"### Response:\n{item['output']}"

    )





if __name__ == "__main__":

    os.makedirs("finetuning", exist_ok=True)



    print(f"Dataset size: {len(TRAINING_DATA)} samples")

    print("\n--- Sample ---")

    print(format_prompt(TRAINING_DATA[0]))



    train_file, val_file = save_dataset(

        TRAINING_DATA,

        "finetuning/dataset.jsonl"

    )

    print("\nDataset preparation complete")

mkdir finetuning

python finetuning/prepare_dataset.py

# finetuning/train_lora.py

"""

LoRA Fine-tuning



Model: microsoft/phi-2 (2.7B, runs on CPU)

LoRA config: rank=8, alpha=32

"""

import os

import json

import torch

from transformers import (

    AutoTokenizer,

    AutoModelForCausalLM,

    TrainingArguments,

    Trainer,

    DataCollatorForLanguageModeling,

)

from peft import LoraConfig, get_peft_model, TaskType

from datasets import Dataset



MODEL_NAME = "microsoft/phi-2"

OUTPUT_DIR = "finetuning/lora_output"

MAX_LENGTH = 512





def load_dataset_from_jsonl(file_path: str) -> Dataset:

    data = []

    with open(file_path, "r", encoding="utf-8") as f:

        for line in f:

            data.append(json.loads(line))

    return Dataset.from_list(data)





def format_prompt(item: dict) -> str:

    if item.get("input"):

        return (

            f"### Instruction:\n{item['instruction']}\n\n"

            f"### Input:\n{item['input']}\n\n"

            f"### Response:\n{item['output']}"

        )

    return (

        f"### Instruction:\n{item['instruction']}\n\n"

        f"### Response:\n{item['output']}"

    )





def main():

    print("=== LoRA Fine-tuning Start ===\n")



    # ── 1. Load model and tokenizer ───────────────────────────

    print(f"Loading model: {MODEL_NAME}")

    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

    tokenizer.pad_token = tokenizer.eos_token



    model = AutoModelForCausalLM.from_pretrained(

        MODEL_NAME,

        dtype=torch.float32,      # float32 for CPU (torch_dtype is deprecated)

        trust_remote_code=True,

    )



    # ── 2. LoRA configuration ─────────────────────────────────

    # r (rank): dimension of low-rank matrices. Larger = more expressive but more memory

    # alpha: scaling factor, typically 2–4× rank

    # target_modules: layers to apply LoRA (usually Attention layers)

    lora_config = LoraConfig(

        task_type=TaskType.CAUSAL_LM,

        r=8,

        lora_alpha=32,

        lora_dropout=0.1,

        target_modules=["q_proj", "v_proj"],

    )



    model = get_peft_model(model, lora_config)

    model.print_trainable_parameters()

    # => trainable params: ~1.3M / total: 2.7B (~0.05%)



    # ── 3. Prepare datasets ───────────────────────────────────

    print("\nLoading datasets...")

    train_dataset = load_dataset_from_jsonl("finetuning/dataset_train.jsonl")

    val_dataset = load_dataset_from_jsonl("finetuning/dataset_val.jsonl")



    def tokenize_function(examples):

        texts = [format_prompt({

            "instruction": inst,

            "input": inp,

            "output": out,

        }) for inst, inp, out in zip(

            examples["instruction"],

            examples["input"],

            examples["output"],

        )]

        tokenized = tokenizer(

            texts,

            truncation=True,

            max_length=MAX_LENGTH,

            padding="max_length",

        )

        tokenized["labels"] = tokenized["input_ids"].copy()

        return tokenized



    train_tokenized = train_dataset.map(tokenize_function, batched=True)

    val_tokenized = val_dataset.map(tokenize_function, batched=True)



    # ── 4. Training configuration ─────────────────────────────

    training_args = TrainingArguments(

        output_dir=OUTPUT_DIR,

        num_train_epochs=3,

        per_device_train_batch_size=1,    # 1 for CPU

        per_device_eval_batch_size=1,

        gradient_accumulation_steps=4,    # effective batch size = 4

        learning_rate=2e-4,

        warmup_steps=10,

        logging_steps=10,

        eval_strategy="epoch",

        save_strategy="epoch",

        load_best_model_at_end=True,

        report_to="none",

        use_cpu=True,                     # CPU mode (no_cuda is deprecated)

    )



    # ── 5. Run training ───────────────────────────────────────

    trainer = Trainer(

        model=model,

        args=training_args,

        train_dataset=train_tokenized,

        eval_dataset=val_tokenized,

        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),

    )



    print("\nTraining started...")

    print("(On CPU, this may take tens of minutes to hours)")

    trainer.train()



    # ── 6. Save model ─────────────────────────────────────────

    model.save_pretrained(OUTPUT_DIR)

    tokenizer.save_pretrained(OUTPUT_DIR)

    print(f"\nModel saved: {OUTPUT_DIR}")





if __name__ == "__main__":

    main()

python finetuning/train_lora.py

# finetuning/inference.py

import torch

from transformers import AutoTokenizer, AutoModelForCausalLM

from peft import PeftModel



MODEL_NAME = "microsoft/phi-2"

LORA_DIR = "finetuning/lora_output"





def load_base_model():

    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

    tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(

        MODEL_NAME,

        dtype=torch.float32,

        trust_remote_code=True,

    )

    return tokenizer, model





def load_finetuned_model():

    tokenizer = AutoTokenizer.from_pretrained(LORA_DIR, trust_remote_code=True)

    tokenizer.pad_token = tokenizer.eos_token

    base_model = AutoModelForCausalLM.from_pretrained(

        MODEL_NAME,

        dtype=torch.float32,

        trust_remote_code=True,

    )

    model = PeftModel.from_pretrained(base_model, LORA_DIR)

    return tokenizer, model





def generate(tokenizer, model, instruction: str, max_new_tokens: int = 200) -> str:

    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"

    inputs = tokenizer(prompt, return_tensors="pt")



    with torch.no_grad():

        outputs = model.generate(

            **inputs,

            max_new_tokens=max_new_tokens,

            temperature=0.7,

            do_sample=True,

            pad_token_id=tokenizer.eos_token_id,

        )



    generated = outputs[0][inputs["input_ids"].shape[1]:]

    return tokenizer.decode(generated, skip_special_tokens=True)





if __name__ == "__main__":

    test_questions = [

        "What is the F1 score?",

        "How do you handle missing values in Pandas?",

    ]



    print("=== Base Model vs Fine-tuned Model Comparison ===\n")



    print("Loading base model...")

    base_tokenizer, base_model = load_base_model()



    print("Loading fine-tuned model...")

    ft_tokenizer, ft_model = load_finetuned_model()



    for question in test_questions:

        print(f"\nQuestion: {question}")

        print("-" * 50)



        base_answer = generate(base_tokenizer, base_model, question)

        print(f"[Base Model]\n{base_answer}")



        ft_answer = generate(ft_tokenizer, ft_model, question)

        print(f"\n[Fine-tuned Model]\n{ft_answer}")

python finetuning/inference.py

✅ Consistency: Same question → same style of answer

✅ Accuracy: No incorrect information

✅ Diversity: Cover a wide variety of question patterns

✅ Conciseness: No unnecessary information

✅ Quantity: Minimum 100 samples, ideally 1000+

cat >> .gitignore << 'EOF'

# Fine-tuning outputs

finetuning/lora_output/

finetuning/*.jsonl

EOF

model.push_to_hub("your-username/my-lora-model")