Writing post-mortem root-cause summaries is time-consuming and inconsistent. Junior SREs miss contributing factors. Senior SREs write summaries that vary in depth and structure. Zero-shot LLMs produce verbose, generic output that does not follow SRE conventions.
Fine-tuning a small model on real incident data produces structured, concise summaries that follow your organisation's format at a fraction of the cost of a large model.
Why This Approach
Diffrent type of approaches and what you get:
Manual SRE writing : Inconsistent, time-consuming, expertise-dependent
Zero-shot large model : Generic format, verbose, high cost per call
Qwen2.5-0.5B fine-tuned : SRE-format outputs, fast, cheap, runs on CPU or consumer GPU
The key advantages of this approach:
- 700-sample training set of real incident timelines mapped to root-cause summaries
- 4-bit quantized LoRA training, runs on a single consumer GPU with 8GB VRAM or more
- Evaluated against a structured rubric covering timeline reference, contributing factors, specific component, and prevention action
- Compared against
qwen3.6-plus:freeandgpt-5.4-nanobaselines
The HuggingFace Model
The fine-tuned adapter is published at: daksh-neo/postmortem-qwen2.5-0.5b-lora
After training, the LoRA weights are saved to models/postmortem-lora/hf_export/ and pushed to HuggingFace.
Quick Start
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # fill in OPENROUTER_API_KEY
export $(cat .env | xargs)
Environment Variables
cp .env.example .env
# Required for baseline evaluation with OpenRouter
OPENROUTER_API_KEY=your_openrouter_api_key_here
OPENROUTER_API_KEY is required only for running baseline evaluations against zero-shot models via OpenRouter. The fine-tuning and local evaluation steps run without it.
Pipeline
The full pipeline runs in four steps:
Each step is independent, you can run baseline evaluation before fine-tuning to establish the gap the fine-tuned model closes, and run evaluation again after to measure the improvement.
Model Configuration
Evaluation Rubric
Every generated summary is scored against a four-criterion rubric. Each criterion carries equal weight:
Pass threshold: 0.60 weighted score or above.
Expected Results
qwen/qwen3.6-plus:free (zero-shot) - 20–35%
openai/gpt-5.4-nano (zero-shot) - 35–50%
Qwen2.5-0.5B (fine-tuned, 3 epochs) - > 60%
The fine-tuned 0.5B model outperforms both zero-shot baselines on rubric compliance because it has been trained specifically on the output format the rubric measures, not on general-purpose tasks.
File Structure
ml_project_0901/
├── scrape_postmortems.py # Data collection
├── baseline.py # Zero-shot baseline via OpenRouter
├── finetune.py # LoRA fine-tuning
├── eval.py # Evaluation + comparison
├── requirements.txt
├── .env.example
├── .gitignore
├── LICENSE
├── CONTRIBUTING.md
├── architecture.excalidraw
├── infographic.svg
├── data/
│ ├── train.jsonl # 700 training examples
│ ├── test_100.jsonl # 100 held-out test examples
│ ├── rubric.json # Scoring rubric
│ └── baseline_results.jsonl
└── models/
└── postmortem-lora/
└── hf_export/ # Push to HuggingFace after training
How I Built This Using NEO
This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.
The requirement was a complete fine-tuning pipeline for a small model on SRE post-mortem data, with data scraping, zero-shot baseline comparison, 4-bit LoRA fine-tuning, and structured rubric-based evaluation. NEO planned, wrote, tested, and verified every file in the repository without human intervention: the data scraper producing 700 training examples and 100 held-out test examples, the baseline evaluator running zero-shot prompts against OpenRouter models, the LoRA fine-tuning script with the full model configuration, the rubric-based evaluator producing the comparison table, and the HuggingFace export pipeline pushing the trained adapter to daksh-neo/postmortem-qwen2.5-0.5b-lora.
How You Can Use and Extend This With NEO
Use it to replace inconsistent manual post-mortem writing in your team.
Train on your own organisation's incident data by replacing data/train.jsonl with your own incident timeline to root-cause summary pairs. The rubric in data/rubric.json can be adapted to match your org's specific post-mortem format the evaluation pipeline measures compliance against whatever criteria you define.
Use the baseline comparison to justify the fine-tuning investment.
Run python baseline.py before fine-tuning to measure what zero-shot models produce on your data. Run python eval.py after fine-tuning to see the improvement. The comparison table gives you a concrete before-and-after that makes the case for domain-specific fine-tuning over general-purpose models.
Use the published adapter directly without retraining.
The fine-tuned LoRA adapter is available at daksh-neo/postmortem-qwen2.5-0.5b-lora on HuggingFace. You can load it directly without running the training pipeline - useful for teams that want to evaluate the output before committing to their own fine-tuning run.
Extend it to other structured generation tasks.
The four-step pipeline - scrape, baseline, fine-tune, evaluate is domain-agnostic. Any task where structured output format matters more than general knowledge is a candidate: alert triage summaries, change request descriptions, deployment notes. Swap the training data and rubric criteria, and the rest of the pipeline runs unchanged.
Final Notes
Zero-shot large models produce verbose, generic post-mortem summaries that do not follow SRE conventions. A fine-tuned 0.5B model trained on 700 domain-specific examples outperforms them on every criterion of the rubric - timeline reference, contributing factors, specific component identification, and concrete prevention actions, while running on a consumer GPU and costing a fraction per call.
The code is at https://github.com/dakshjain-1616/postmortem-finetune
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code