Building a Production-Grade Observability Platform with LGTM Stack, DORA Metrics & SLOs

GitHub Repository:

📸 [Screenshot: docker compose ps showing all 9 services Up]

Part 1: Deploying the Full LGTM Stack

Docker Compose — the complete stack

CODE Kopieren

# docker-compose.yml

version: "3.8"



networks:

  observability:

    driver: bridge



volumes:

  prometheus_data:

  loki_data:

  tempo_data:

  grafana_data:



services:

  prometheus:

    image: prom/prometheus:v2.51.0

    container_name: prometheus

    restart: unless-stopped

    command:

      - "--config.file=/etc/prometheus/prometheus.yml"

      - "--storage.tsdb.path=/prometheus"

      - "--storage.tsdb.retention.time=30d"

      - "--web.enable-lifecycle"

      - "--web.enable-remote-write-receiver"

    volumes:

      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro

      - ./alerts:/etc/prometheus/alerts:ro

      - prometheus_data:/prometheus

    ports:

      - "9090:9090"

    networks:

      - observability



  loki:

    image: grafana/loki:2.9.7

    container_name: loki

    restart: unless-stopped

    command: -config.file=/etc/loki/loki-config.yaml

    volumes:

      - ./config/loki-config.yaml:/etc/loki/loki-config.yaml:ro

      - loki_data:/loki

    ports:

      - "3100:3100"

    networks:

      - observability



  tempo:

    image: grafana/tempo:2.4.1

    container_name: tempo

    restart: unless-stopped

    command: -config.file=/etc/tempo/tempo.yaml

    volumes:

      - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro

      - tempo_data:/var/tempo

    ports:

      - "3200:3200"

      - "4317:4317"

      - "4318:4318"

    networks:

      - observability



  grafana:

    image: grafana/grafana:10.4.2

    container_name: grafana

    restart: unless-stopped

    environment:

      - GF_SECURITY_ADMIN_PASSWORD=admin

      - GF_USERS_ALLOW_SIGN_UP=false

      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor

    volumes:

      - grafana_data:/var/lib/grafana

      - ./grafana/provisioning:/etc/grafana/provisioning:ro

      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro

    ports:

      - "3000:3000"

    networks:

      - observability



  alertmanager:

    image: prom/alertmanager:v0.27.0

    container_name: alertmanager

    restart: unless-stopped

    volumes:

      - ./config/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro

      - ./config/slack.tmpl:/etc/alertmanager/slack.tmpl:ro

    ports:

      - "9093:9093"

    networks:

      - observability



  node-exporter:

    image: prom/node-exporter:v1.7.0

    container_name: node-exporter

    restart: unless-stopped

    volumes:

      - /proc:/host/proc:ro

      - /sys:/host/sys:ro

      - /:/rootfs:ro

    ports:

      - "9100:9100"

    networks:

      - observability



  blackbox-exporter:

    image: prom/blackbox-exporter:v0.25.0

    container_name: blackbox-exporter

    restart: unless-stopped

    volumes:

      - ./config/blackbox.yml:/etc/blackbox_exporter/config.yml:ro

    ports:

      - "9115:9115"

    networks:

      - observability



  pushgateway:

    image: prom/pushgateway:v1.7.0

    container_name: pushgateway

    restart: unless-stopped

    ports:

      - "9091:9091"

    networks:

      - observability



  otel-collector:

    image: otel/opentelemetry-collector-contrib:0.98.0

    container_name: otel-collector

    restart: unless-stopped

    command: ["--config=/etc/otel/otel-collector.yaml"]

    volumes:

      - ./config/otel-collector.yaml:/etc/otel/otel-collector.yaml:ro

    ports:

      - "4319:4317"

      - "4320:4318"

      - "8888:8888"

    networks:

      - observability

One command to bring everything up

CODE Kopieren

docker compose up -d

Infrastructure as Code — non-negotiable

Every configuration file is version-controlled. Nothing is configured through a UI:

CODE Kopieren

config/

├── prometheus.yml        # Scrape configs + recording rules

├── alertmanager.yml      # Route trees + inhibition rules

├── loki-config.yaml      # Log ingestion + 30d retention

├── tempo.yaml            # Trace storage + 30d retention

├── otel-collector.yaml   # Trace and log pipeline

└── blackbox.yml          # HTTP + SSL probe modules



alerts/

├── infrastructure.yml    # CPU, memory, disk, host down

├── slo-burnrate.yml      # Multi-window burn rate alerts

└── cicd.yml              # DORA threshold alerts



grafana/

├── provisioning/         # Datasource + dashboard discovery

└── dashboards/           # 5 JSON dashboards

Prometheus scrape configuration

CODE Kopieren

# config/prometheus.yml

global:

  scrape_interval: 15s

  evaluation_interval: 15s



rule_files:

  - /etc/prometheus/alerts/infrastructure.yml

  - /etc/prometheus/alerts/slo-burnrate.yml

  - /etc/prometheus/alerts/cicd.yml



scrape_configs:

  - job_name: node-exporter

    scrape_interval: 15s

    static_configs:

      - targets: ["node-exporter:9100"]



  - job_name: blackbox-http

    metrics_path: /probe

    params:

      module: [http_2xx]

    static_configs:

      - targets:

          - http://grafana:3000

          - http://prometheus:9090/-/healthy

          - http://loki:3100/ready

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: blackbox-exporter:9115



  - job_name: pushgateway

    honor_labels: true

    static_configs:

      - targets: ["pushgateway:9091"]

Retention periods:

• Prometheus metrics: 30 days (--storage.tsdb.retention.time=30d)


• Loki logs: 30 days (retention_period: 30d in loki-config.yaml)


• Tempo traces: 30 days (block_retention: 720h in tempo.yaml)

📸 [Screenshot: SLO & Error Budget dashboard showing gauges and burn rate]

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

DORA metrics answer: "Is our team getting better or worse at delivering software safely?"

Metric	Business impact
Deployment Frequency	How often value reaches users
Lead Time for Changes	How quickly a bug fix ships
Change Failure Rate	Cost of broken deployments
Mean Time to Restore	Duration of user impact during incidents

DORA benchmarks

Metric	Elite	High	Medium	Low
Deploy frequency	Multiple/day	Weekly	Monthly	< Monthly
Lead time	< 1 hour	< 1 day	1d–1w	> 1 week
CFR	< 5%	5–10%	10–15%	> 15%
MTTR	< 1 hour	< 1 day	1d–1w	> 1 week

GitHub Actions pushing DORA metrics to Pushgateway

CODE Kopieren

# .github/workflows/deploy.yml

jobs:

  deploy:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v4



      - name: Record deploy start time

        id: timing

        run: echo "start_ts=$(date +%s)" >> $GITHUB_OUTPUT



      - name: Build and deploy

        run: |

          echo "Your actual build and deploy steps here"



      - name: Push DORA metrics on success

        if: success()

        run: |

          LEAD_TIME=$(( $(date +%s) - ${{ steps.timing.outputs.start_ts }} ))

          WORKFLOW="${{ github.workflow }}"



          # Deployment counter

          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"

          deployment_total{status="success",workflow="${WORKFLOW}"} 1

          EOF



          # Lead time

          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"

          deployment_lead_time_seconds{workflow="${WORKFLOW}"} ${LEAD_TIME}

          EOF



      - name: Push DORA metrics on failure

        if: failure()

        run: |

          WORKFLOW="${{ github.workflow }}"

          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"

          deployment_total{status="failure",workflow="${WORKFLOW}"} 1

          EOF

DORA recording rules in Prometheus

CODE Kopieren

groups:

  - name: cicd.recording_rules

    rules:

      # Deployment frequency

      - record: dora:deployment_frequency:rate24h

        expr: sum(increase(deployment_total[24h])) by (workflow)



      # Change Failure Rate = failed / total over 7 days

      - record: dora:change_failure_rate:ratio7d

        expr: |

          sum(increase(deployment_total{status="failure"}[7d])) by (workflow)

          /

          sum(increase(deployment_total[7d])) by (workflow)



      # Mean Time to Restore

      - record: dora:mttr:avg7d

        expr: avg_over_time(deployment_restore_time_seconds[7d])

Toil identified and automated

Toil 1 — Manual alert acknowledgement. Engineers read a Slack alert, open a browser, navigate to Grafana, search for the relevant dashboard. Automation: every alert payload includes a direct link to the exact dashboard. Saves 2–3 minutes per alert.

Toil 2 — Certificate renewal reminders. SSL expiry tracked via calendar reminders. Automation: Blackbox Exporter monitors SSL expiry continuously. SSLCertExpiringSoon alert fires 14 days before expiry automatically.

📸 [Screenshot: Node Exporter dashboard with live CPU and memory data]

Dashboard 2 — Blackbox Exporter

External probing: uptime/downtime timeline, HTTP response time, SSL certificate expiry countdown, probe success rate. This dashboard answers "what is the user experiencing?" rather than "what is the server doing?" — a critical distinction.

📸 [Screenshot: SLO dashboard with error budget gauge]

Dashboard 5 — Unified Observability (the most important)

This is the dashboard that makes the entire stack worth building.

A user sees a spike in the error rate panel → clicks through to Loki → sees error logs from that exact time window → clicks the trace ID link → Tempo opens the waterfall → identifies exactly which service, endpoint, and span caused the failure.

This drill-down — metric spike → correlated logs → causing trace — is what separates observability from monitoring.

CODE Kopieren

Monitoring: "Something is wrong"

Observability: "Here is exactly why, where, and when"

📸 [Screenshot: Loki logs panel with clickable trace IDs]

Part 6: The Alerting System

All alert rules are version-controlled

Zero alert rules live in Grafana. Every rule is in a .yml file under alerts/.

Infrastructure alerts

CODE Kopieren

# alerts/infrastructure.yml

groups:

  - name: infrastructure.rules

    rules:

      # Recording rules — pre-compute SLIs

      - record: sli:node_cpu_saturation

        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))



      - record: sli:node_memory_saturation

        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)



      # CPU alerts

      - alert: HighCPUWarning

        expr: sli:node_cpu_saturation > 0.80

        for: 5m

        labels:

          severity: warning

        annotations:

          summary: "High CPU usage on {{ $labels.instance }}"

          description: "CPU is {{ $value | humanizePercentage }} (threshold: 80%)"

          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"

          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"



      - alert: HighCPUCritical

        expr: sli:node_cpu_saturation > 0.90

        for: 10m

        labels:

          severity: critical

        annotations:

          summary: "Critical CPU on {{ $labels.instance }}"

          description: "CPU is {{ $value | humanizePercentage }} for 10+ minutes"

          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"

          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"



      # Host down — Blackbox probe fails for 2 minutes

      - alert: HostDown

        expr: probe_success == 0

        for: 2m

        labels:

          severity: critical

        annotations:

          summary: "Host {{ $labels.instance }} is down"

          description: "Blackbox probe failed for 2+ consecutive minutes"

          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/host-down.md"

Burn rate alerting — how it reduces alert fatigue

Traditional threshold alerting fires whenever a metric crosses a line. This produces alert storms — dozens of notifications for a single incident. Teams learn to ignore them.

Burn rate alerting answers a different question: "At this rate of failure, how long until our error budget is exhausted?"

Two alerts replace an entire category of noise:

CODE Kopieren

# alerts/slo-burnrate.yml

  - name: slo.alerts

    rules:

      # Fast burn — act immediately

      # 14.4x means 2% of monthly budget gone in 1 hour

      - alert: SLOAvailabilityFastBurn

        expr: slo:availability:burn_rate1h > 14.4

        for: 2m

        labels:

          severity: critical

        annotations:

          summary: "Fast error budget burn — act immediately"

          description: >

            Burn rate is {{ $value | humanize }}x. At this rate,

            2% of the 30-day budget will be consumed in 1 hour.

          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-fast-burn.md"



      # Slow burn — investigate before it escalates

      # 5x means 5% of monthly budget gone in 6 hours

      - alert: SLOAvailabilitySlowBurn

        expr: slo:availability:burn_rate6h > 5

        for: 15m

        labels:

          severity: warning

        annotations:

          summary: "Slow error budget burn — investigate soon"

          description: >

            Burn rate is {{ $value | humanize }}x over 6h.

            5% of the 30-day budget will be consumed in 6 hours.

          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-slow-burn.md"

Alertmanager routing and inhibition

CODE Kopieren

# config/alertmanager.yml

route:

  receiver: slack-default

  group_by: [alertname, severity, instance]

  group_wait: 30s

  group_interval: 5m

  repeat_interval: 4h

  routes:

    - match:

        severity: critical

      receiver: slack-critical

      group_wait: 10s

      repeat_interval: 4h



inhibit_rules:

  # When host is completely down, suppress CPU/memory/latency noise

  - source_match:

      alertname: HostDown

    target_match_re:

      alertname: "HighCPU.*|HighMemory.*|HighLatency.*|DiskSpace.*"

    equal: [instance]



  # Critical suppresses warning for same alert on same host

  - source_match:

      severity: critical

    target_match:

      severity: warning

    equal: [alertname, instance]

Structured Slack payload — plain text is not acceptable

Every alert in #all-hng-alerts includes alert name, severity, host, metric value, Grafana link, and runbook link.

CODE Kopieren

# config/slack.tmpl

{{ define "slack.title" -}}

[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}

{{- end }}



{{ define "slack.body" -}}

{{ range .Alerts }}

*Alert:* {{ .Labels.alertname }}

*Severity:* {{ .Labels.severity | toUpper }}

*Status:* {{ if eq $.Status "resolved" }}✅ RESOLVED{{ else }}🔥 FIRING{{ end }}

*Host:* {{ .Labels.instance }}

*Summary:* {{ .Annotations.summary }}



*Links:*

• <{{ .Annotations.dashboard_url }}|📊 Grafana Dashboard>

• <{{ .Annotations.runbook_url }}|📖 Runbook>



*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}

{{ end }}

{{- end }}

📸 [Screenshot: Slack showing RESOLVED alert]

Part 7: Runbooks and Incident Management

A runbook for every alert

Every alert links directly to its runbook. An engineer woken at 3am should be able to follow it to resolution without searching.

Each runbook answers six questions:

CODE Kopieren

# Runbook: High CPU Usage



## What is this alert?

HighCPUWarning fires when CPU exceeds 80% for 5+ minutes.



## Likely cause

1. Traffic spike

2. Runaway process

3. Post-deployment regression



## First 3 investigation steps

1. Check running processes:

bash

top -bn1 | head -20

docker stats --no-stream

CODE Kopieren

2. Correlate with traffic on Unified Observability dashboard

3. Check recent deployments in GitHub Actions



## Resolution

- Runaway process: kill -9 <PID>

- Traffic spike: scale horizontally

- Deployment regression: roll back



## Roll back when?

If CPU spike started within 30 minutes of a deployment

and correlates with increased error rate.



## Escalation

Senior engineer if unresolved after 20 minutes.

Blameless Post-Incident Review

We documented a simulated incident where a missing environment variable caused 35% of requests to return 503 for 47 minutes.

Timeline:

Time	Event
14:18	Deployment triggered
14:23	503 responses begin
14:29	SLOAvailabilityFastBurn fires (6-min detection lag)
14:36	Trace ID in Loki → Tempo reveals config read failure
14:40	Root cause identified: missing DATABASE_URL env var
14:45	Rollback initiated
15:10	Error rate returns to baseline

Root cause: New environment variable added to code but not to docker-compose.yml.

Detection gap: 6-minute lag between incident start and alert firing. Action item: reduce fast-burn for: clause from 2m to 1m.

Action items:

Action	Owner	Due
Add post-deploy smoke test	DevOps	3 days
Add env var validation to entrypoint	App dev	5 days
Reduce fast-burn for: clause to 1m	DevOps	1 day

This review is blameless — we focus on systems and processes, not individuals.

Part 8: Game Day Results

Scenario 1 — Deployment Failure

Added exit 1 to the GitHub Actions workflow and pushed. The workflow failed and pushed deployment_total{status="failure"} to the Pushgateway. CICDDeploymentFailed fired in Slack within 2 minutes. DORA dashboard showed CFR increase. Immediately reverted.

📸 [Screenshot: CICDDeploymentFailed in Slack]

Scenario 2 — Latency Injection

Injected 600ms network latency:

CODE Kopieren

sudo tc qdisc add dev ens5 root netem delay 600ms

HighLatencyWarning fired confirming the alerting pipeline for latency SLO breaches works end-to-end.

CODE Kopieren

# Remove latency

sudo tc qdisc del dev ens5 root

RESOLVED message confirmed recovery detection works.

📸 [Screenshot: HighLatencyWarning in Slack]

📸 [Screenshot: Prometheus alerts page showing Warning firing]

📸 [Screenshot: Node Exporter dashboard with CPU spike at 92%]

📸 [Screenshot: RESOLVED in Slack]

Key Learnings

1. Observability is not monitoring.

Monitoring tells you something is wrong. Observability tells you why, where, and when — without needing to SSH into a server.

2. SLOs make reliability decisions objective.

"Is this deployment safe?" is subjective. "Do we have 100 minutes of error budget remaining?" is objective. SLOs turn reliability from a conversation into a measurement.

3. Burn rate alerting eliminates alert fatigue.

Two burn rate alerts replaced what would have been dozens of threshold alerts during our Game Day scenarios. Engineers respond to meaningful signals, not noise.

4. DORA metrics connect engineering to business.

High MTTR isn't just a technical problem — it's lost revenue per minute. Low deployment frequency isn't just slow — it's delayed value delivery. DORA makes this explicit.

5. Everything as code is non-negotiable.

Every dashboard, alert rule, and config that lives only in a UI is technical debt. When the server dies, you want to run docker compose up -d and have everything back — not spend three hours recreating dashboards from memory.

Conclusion

The MeetMind Observability Platform demonstrates that production-grade observability is achievable without managed services. The LGTM stack provides the full observability triad — metrics, logs, and traces — with correlation between all three. SLOs convert vague reliability goals into measurable targets. DORA metrics connect daily engineering decisions to business outcomes. Burn rate alerting replaces alert storms with two meaningful signals.

The entire platform deploys with one command. Every component is version-controlled. Every alert links to a runbook. Every metric spike links to correlated logs and traces.

GitHub Repository: https://github.com/AirFluke/meetmind-observability

Built by Team MeetMind for HNG DevOps Track Stage 6

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen