Building a Production-Grade Observability Platform with the LGTM Stack, DORA Metrics & SLOs

GitHub Repository:

📸 [Screenshot: All 9 services showing running status]

Part 1: Deploying the Full LGTM Stack as Systemd Services

Why systemd over Docker?

Running services as native systemd units means:

• No container runtime dependency


• Services start on boot automatically


• Logs go directly to journald — journalctl -u prometheus -f
  


• Standard Linux process management — systemctl start/stop/restart/status
  


• No networking complexity — all services talk via localhost

One-command deployment

CODE Kopieren

git clone https://github.com/AirFluke/meetmind-observability.git

cd meetmind-observability

sudo SLACK_WEBHOOK=https://hooks.slack.com/services/YOUR/WEBHOOK bash install.sh

The install script handles everything automatically:

1. Installs system dependencies


2. Downloads all binaries from GitHub releases


3. Creates dedicated system users for each service


4. Copies configs to /etc/
   


5. Creates data directories in /var/lib/
   


6. Installs systemd unit files to /etc/systemd/system/
   


7. Enables and starts all services

Systemd unit files — the core of the deployment

Each service has a unit file that defines how it runs. Here is Prometheus as an example:

CODE Kopieren

# systemd/prometheus.service

[Unit]

Description=Prometheus Metrics Server

Documentation=https://prometheus.io/docs

After=network-online.target

Wants=network-online.target



[Service]

Type=simple

User=prometheus

Group=prometheus

ExecStart=/usr/local/bin/prometheus \

  --config.file=/etc/prometheus/prometheus.yml \

  --storage.tsdb.path=/var/lib/prometheus \

  --storage.tsdb.retention.time=30d \

  --web.enable-lifecycle \

  --web.enable-remote-write-receiver \

  --web.listen-address=0.0.0.0:9090



Restart=on-failure

RestartSec=5s

LimitNOFILE=65536



[Install]

WantedBy=multi-user.target

Restart=on-failure with RestartSec=5s is the systemd equivalent of Docker's restart: unless-stopped. Every service has this.

Checking service status

CODE Kopieren

# Check all 9 at once

sudo bash scripts/status.sh



# Check individual service

sudo systemctl status prometheus



# Follow logs in real time

journalctl -u prometheus -f

journalctl -u grafana-server -f

journalctl -u loki -f

File layout on the server

CODE Kopieren

/usr/local/bin/          ← all binaries

  prometheus, loki, tempo, alertmanager,

  node_exporter, blackbox_exporter,

  pushgateway, otelcol



/etc/                    ← all configs

  prometheus/prometheus.yml

  alertmanager/alertmanager.yml

  alertmanager/slack.tmpl

  loki/loki-config.yaml

  tempo/tempo.yaml

  otelcol/otel-collector.yaml

  blackbox_exporter/config.yml



/var/lib/                ← all data (30d retention)

  prometheus/

  loki/

  tempo/



/etc/systemd/system/     ← unit files

  prometheus.service

  loki.service

  tempo.service

  ... (9 total)

Retention periods

• Prometheus metrics: 30 days (--storage.tsdb.retention.time=30d)


• Loki logs: 30 days (retention_period: 30d)


• Tempo traces: 30 days (block_retention: 720h)

Infrastructure as Code — non-negotiable

Every configuration file is version-controlled in the repository:

CODE Kopieren

meetmind-observability/

├── install.sh                  ← one-command deploy

├── uninstall.sh                ← clean teardown

├── scripts/status.sh           ← check all services

├── systemd/                    ← 9 unit files

├── config/                     ← all service configs

├── alerts/                     ← alert rules (.yml)

├── grafana/dashboards/         ← 5 JSON dashboards

├── grafana/provisioning/       ← datasource config

└── runbooks/                   ← one .md per alert

Nothing requires manual configuration to reproduce. Clone the repo, run the install script, and the entire platform is up.

📸 [Screenshot: SLO & Error Budget dashboard]

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

Metric	What it measures	Business impact
Deployment Frequency	How often value reaches users	Faster delivery
Lead Time for Changes	Commit to production	Bug fix speed
Change Failure Rate	Broken deployments	Cost of poor quality
Mean Time to Restore	Duration of incidents	User impact per outage

DORA benchmarks

Metric	Elite	High	Medium	Low
Deploy frequency	Multiple/day	Weekly	Monthly	< Monthly
Lead time	< 1 hour	< 1 day	1d–1w	> 1 week
CFR	< 5%	5–10%	10–15%	> 15%
MTTR	< 1 hour	< 1 day	1d–1w	> 1 week

GitHub Actions pushing DORA metrics to Pushgateway

CODE Kopieren

# .github/workflows/deploy.yml

jobs:

  deploy:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v4



      - name: Record deploy start time

        id: timing

        run: echo "start_ts=$(date +%s)" >> $GITHUB_OUTPUT



      - name: Build and deploy

        run: echo "Your deploy steps here"



      - name: Push metrics on success

        if: success()

        run: |

          LEAD_TIME=$(( $(date +%s) - ${{ steps.timing.outputs.start_ts }} ))

          WORKFLOW="${{ github.workflow }}"



          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"

          deployment_total{status="success",workflow="${WORKFLOW}"} 1

          deployment_lead_time_seconds{workflow="${WORKFLOW}"} ${LEAD_TIME}

          EOF



      - name: Push metrics on failure

        if: failure()

        run: |

          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"

          deployment_total{status="failure",workflow="${{ github.workflow }}"} 1

          EOF

Toil identified and automated

Toil 1 — Manual alert acknowledgement.

Engineers read a Slack alert, open a browser, navigate to Grafana, search for the relevant dashboard. Automation: every alert payload includes a direct link to the exact dashboard. Saves 2–3 minutes per alert.

Toil 2 — Certificate renewal reminders.

SSL expiry tracked via calendar reminders. Automation: Blackbox Exporter monitors SSL expiry continuously. SSLCertExpiringSoon alert fires 14 days before expiry automatically.

📸 [Screenshot: Node Exporter dashboard with live data]

Dashboard 2 — Blackbox Exporter

External probing: uptime/downtime timeline, HTTP response time, SSL certificate expiry countdown, probe success rate.

📸 [Screenshot: DORA dashboard]

Dashboard 4 — SLO & Error Budget

SLI vs SLO gauges, error budget remaining coloured by urgency, burn rate time series with fast/slow thresholds, compliance history.

📸 [Screenshot: Unified dashboard]

📸 [Screenshot: Slack showing firing alert with full structured payload]

📸 [Screenshot: GitHub Actions showing red failed run]

📸 [Screenshot: HighLatencyWarning in Slack]

📸 [Screenshot: Prometheus alerts page showing Warning firing]

📸 [Screenshot: HighCPUWarning in Slack]

Deploy it yourself:

CODE Kopieren

git clone https://github.com/AirFluke/meetmind-observability.git

cd meetmind-observability

sudo SLACK_WEBHOOK=https://hooks.slack.com/services/YOUR/WEBHOOK bash install.sh

Built by Team MeetMind for HNG DevOps Track Stage 6

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen