Observability Practices: A Hands-On Guide with Prometheus and Grafana

What is Observability?

Observability is the ability to understand the internal state of a system just by looking at its external outputs. Unlike traditional monitoring, which tells you whether something is wrong, observability helps you understand why it's wrong.

It's built on three pillars:

• 
  Logs: discrete, timestamped events (e.g., "User 123 logged in").


• 
  Metrics: numeric measurements over time (e.g., request latency, memory usage).


• 
  Traces: the path a request takes through a distributed system.

In this article, I'll walk through a real example: instrumenting a Node.js API with Prometheus for metrics collection and Grafana for visualization. Both tools are free, open-source, and widely used in production.

Why This Stack?

Prometheus and Grafana are a great starting point because:

• They're free and open-source.


• Prometheus uses a pull-based model, scraping metrics from your app at intervals.


• Grafana turns those metrics into readable dashboards.


• The combination is an industry standard, used alongside commercial tools like Datadog or New Relic.

Step 1: Instrumenting a Node.js App

We'll build a small Express API and expose custom metrics using prom-client, the official Prometheus client library for Node.js.

npm install express prom-client

// server.js

const express = require('express');

const client = require('prom-client');



const app = express();

const register = new client.Registry();



// Collect default Node.js metrics (CPU, memory, event loop lag, etc.)

client.collectDefaultMetrics({ register });



// Custom metric: counts total HTTP requests by route and status code

const httpRequestCounter = new client.Counter({

  name: 'http_requests_total',

  help: 'Total number of HTTP requests',

  labelNames: ['method', 'route', 'status_code'],

});

register.registerMetric(httpRequestCounter);



// Custom metric: measures request duration

const httpRequestDuration = new client.Histogram({

  name: 'http_request_duration_seconds',

  help: 'Duration of HTTP requests in seconds',

  labelNames: ['method', 'route', 'status_code'],

  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],

});

register.registerMetric(httpRequestDuration);



// Middleware to track every request

app.use((req, res, next) => {

  const end = httpRequestDuration.startTimer();

  res.on('finish', () => {

    const labels = { method: req.method, route: req.path, status_code: res.statusCode };

    httpRequestCounter.inc(labels);

    end(labels);

  });

  next();

});



// Sample endpoints

app.get('/', (req, res) => {

  res.send('Welcome to the Observability Demo API');

});



app.get('/slow', async (req, res) => {

  // Simulate a slow endpoint

  await new Promise((resolve) => setTimeout(resolve, Math.random() * 2000));

  res.send('This endpoint is intentionally slow');

});



app.get('/error', (req, res) => {

  res.status(500).send('Something went wrong');

});



// Expose metrics endpoint for Prometheus to scrape

app.get('/metrics', async (req, res) => {

  res.set('Content-Type', register.contentType);

  res.end(await register.metrics());

});



app.listen(3000, () => console.log('Server running on http://localhost:3000'));

This exposes a /metrics endpoint in the Prometheus text format, which looks something like this:

http_requests_total{method="GET",route="/",status_code="200"} 12

http_request_duration_seconds_bucket{le="0.5",method="GET",route="/slow"} 3

Step 2: Configuring Prometheus

Prometheus needs to know where to scrape metrics from. Here's a minimal prometheus.yml:

global:

  scrape_interval: 5s



scrape_configs:

  - job_name: 'node-app'

    static_configs:

      - targets: ['localhost:3000']

Run Prometheus with Docker:

docker run -d -p 9090:9090 -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Now visit http://localhost:9090 and query http_requests_total — you'll see live data flowing in from your app.

Step 3: Visualizing with Grafana

Run Grafana alongside Prometheus:

docker run -d -p 3001:3000 grafana/grafana

1. Open http://localhost:3001 (default login: admin/admin).


2. Add Prometheus as a data source, pointing to http://host.docker.internal:9090.


3. Create a new dashboard with panels for:
   
   
   
   
   • Request rate: rate(http_requests_total[1m])
     
   
   
   • Error rate: rate(http_requests_total{status_code="500"}[1m])
     
   
   
   • p95 latency: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
     
   
   
   
   

Within minutes, you have a live dashboard showing traffic, errors, and latency — the core signals every team needs to detect issues before users complain.

Why This Matters in Practice

Imagine your /slow endpoint starts timing out under load. Without observability, you'd only find out when a user complains. With metrics like p95 latency and error rate visible on a dashboard (and alerts configured on top of them), your team can catch the regression within minutes of deployment — often before it affects most users.

This same pattern — instrument, expose, scrape, visualize — applies whether you're using Prometheus/Grafana, Datadog, New Relic, or Azure Monitor. The tools differ, but the principle is the same: you can't fix what you can't see.

Key Takeaways

• Observability goes beyond monitoring: it helps you answer why, not just what.


• Instrumenting code with metrics (counters, histograms) is a low-effort, high-value practice.


• Prometheus + Grafana is a free, production-grade way to get started.


• The same principles apply across any observability platform.