Prometheus Fundamentals (Lesson-01)

1. Getting Started

1.1 Introduction

We'll start with the basics, discussing below things,

• What Prometheus is and how it functions


• Installation and configuration.


• Set up your own Prometheus server.


• Examine the Prometheus data model, its storage methods, and the query language for interacting with data.


• Visualization techniques for representing Prometheus metric data. Various data collection methods.


• Alerting mechanisms.


• We'll touch upon advanced topics like high availability, federation, and security considerations in Prometheus.

2. Prometheus Basics

2.1 What is Prometheus

Prometheus is an essential open-source tool for monitoring and alerting. Prometheus plays a critical role in gathering data about applications and systems, enabling visualization and real-time alerts based on this data. Understanding monitoring is crucial in managing a robust DevOps infrastructure; it allows you to quickly and easily comprehend what's happening within your systems.

Prometheus excels in the metric collection, storing crucial data about systems and applications on its server. This data can then be visualized through various means, such as dashboards with graphs and charts, providing actionable insights.

Moreover, Prometheus is adept at alerting, where it can react in real-time to changes in data. For instance, if a server goes down, Prometheus can issue an alert, notifying you to address the issue.

Use Cases

Metric collection: Collect Important metrics about your systems and applications in one place.

Visualization: Build dashboards that provide an overview of the health of your systems.

Alerting: Receive an email when something is broken.

As for its technical background, Prometheus is primarily written in Go, with some components in Java, Python, and Ruby. It's open-source under the Apache 2.0 license, meaning it's free for varied uses. Originally developed by Matt T. Proud and Julius Volz and sponsored by SoundCloud, it now thrives as a project maintained by various individuals and organizations.

For comprehensive information and full documentation on Prometheus, you can visit prometheus.io.

2.2 Prometheus Architecture

Prometheus is composed of various components, but the two most fundamental are the Prometheus server and exporters.

The Prometheus server acts as a central hub that gathers and stores metrics. Exporters, on the other hand, are agents that expose data about systems and applications for the server to collect.

Prometheus server: A central server that gathers metrics and makes them available.

Exporters: Agents that expose data about systems and applications for collection by the Prometheus server.

Prometheus Pull Model

A key aspect of Prometheus is its pull model for metric collection, this means the Prometheus server pulls metric data from exporters - agents do not push data to the Prometheus server.

Prometheus Architecture

• Prometheus Server: Collects metrics data.


• Exporters: Provide metric data for Prometheus to consume.


• Client Libraries: Easily turn your custom application into an exporter that exposes metrics in a format Prometheus can consume.


• Prometheus Pushgatewway: Allows pushing metrics to Prometheus for certain specific use cases.


• Alertmanager: Sends alerts triggered by metric data.


• Visualization Tools: Provide useful ways to view metric data. (Not necessary)

Inside the Prometheus server, there are multiple components, including storage for metric data. The server is the core of the entire Prometheus ecosystem. Alongside this, Prometheus employs client libraries for different programming languages, allowing the creation of custom exporters. For instance, a Java application can be turned into an exporter using the Prometheus Java client library, enabling the server to pull metrics from it.

Prometheus also includes a Pushgateway for scenarios where a pull model is not feasible, like short-lived batch jobs. This component acts as a middleman, allowing these jobs to push metrics to it, which Prometheus can then pull. Additionally, the Prometheus Alertmanager is responsible for sending out alerts triggered by metric data.

Visualization tools are another critical component of Prometheus, offering various ways to interact with and make decisions based on metric data. These tools can be part of Prometheus or external, like Grafana, providing charts and graphs for data analysis

3. Installation and Configuration

3.1 Intstallation

In this lesson, we guide you through the process of building a Prometheus server. There are several installation options for Prometheus, including using precompiled binaries from prometheus.io, building from source code on GitHub, or running it in a Docker container.

Step 01:

The first step involves creating a Prometheus user and group using the useradd command.

$ sudo useradd -M -r -s /bin/false prometheus

Next, directories for storing configuration files and Prometheus data are created: /etc/prometheus and /var/lib/prometheus. Following this, the Prometheus archive is downloaded from prometheus.io and extracted.

$ sudo mkdir /etc/prometheus /var/lib/prometheus

Step 02:

$ wget https://github.com/prometheus/prometheus/releases/download/v2.48.1/prometheus-2.48.1.linux-amd64.tar.gz

$ sudo tar -xzvf prometheus-2.48.1.linux-amd64.tar.gz

The extracted binaries, prometheus and promtool, are then copied to /usr/local/bin/, and ownership of these binaries is set to the Prometheus user and group.

$ sudo cp prometheus-2.48.1.linux-amd64/{prometheus,promtool} /usr/local/bin/

Set the permission to 'prometheus'

[root@ip ~]# ll /usr/local/bin/

total 238496

-rwxr-xr-x. 1 root root 125066649 Dec 29 17:40 prometheus

-rwxr-xr-x. 1 root root 119151581 Dec 29 17:40 promtool

$ sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}

Now, copy the 'consoles' and 'console_libraries' files to /etc/prometheus

$ sudo cp prometheus-2.48.1.linux-amd64/{consoles,console_libraries} /etc/prometheus

The next steps involve setting up the configuration file, prometheus.yml, and copying it to /etc/prometheus/

$ sudo cp prometheus-2.48.1.linux-amd64/prometheus.yml /etc/prometheus/

Set the permission prometheus in /etc/prometheus dir and /var/lib/prometheus

$ sudo chown -R prometheus:prometheus /etc/prometheus



$ sudo chown -R prometheus:prometheus /var/lib/prometheus

Step 03:

After ensuring proper file ownership, we test running Prometheus in the foreground to confirm it's ready to receive web requests. To make Prometheus more manageable, it's turned into a systemd service by creating a prometheus.service unit file, detailing how the service should run.

Test:

$ prometheus --config.file=/etc/prometheus/prometheus.yml

If the test succeeds you can see "Server is ready to receive web requests." in the logs.

Create the unit file and the below content.

$ sudo vim /etc/systemd/system/prometheus.service

[Unit]

Description=Prometheus Time Series Collection and Processing Server

Wants=network-online.target

After=network-online.target



[Service]

User=prometheus

Group=prometheus

Type=simple

ExecStart=/usr/local/bin/prometheus \

 --config.file /etc/prometheus/prometheus.yml \

 --storage.tsdb.path /var/lib/prometheus/ \

 --web.console.templates=/etc/prometheus/consoles \

 --web.console.libraries=/etc/prometheus/console_libraries

[Install]

WantedBy=multi-user.target

Start and enable the Prometheus service:

$ sudo systemctl daemon-reload



$ sudo systemctl enable --now prometheus

Make an HTTP request to Prometheus to verify it is able to respond:

$ curl localhost:9090

You can also access Prometheus in a browser using the server's public IP address: http://PROMETHEUS_SERVER_PUBLIC_IP:9090

3.2 Configuring an Exporter

We focus on configuring a Prometheus exporter, specifically the Node Exporter, which is an essential component in the Prometheus architecture. The Node Exporter is a type of Prometheus exporter designed to run on *NIX machines, collecting various system metrics. We'll be setting up a simple Linux server, installing the Node Exporter program, and configuring it as a systemd service. This setup allows the exporter to expose metrics about the machine, which the Prometheus server can then scrape and collect.

Once the Node Exporter is set up on our Linux server, the next crucial step is to configure the scrape_config in our Prometheus server's configuration file (prometheus.yml). The scrape_config section in the Prometheus configuration file specifies the list of targets for Prometheus to scrape. This is where we tell Prometheus where to find the exporters and how to connect to them to retrieve metric data.

For those wishing to follow along with the demonstration, you'll need to set up an additional Linux server.

Once this server is set up, the Node Exporter binary is downloaded, installed, and configured as a systemd service to ensure it's always running in the background.

Step 01:

Create a user called "node_exporter":

$ sudo useradd -M -r -s /bin/false node_exporter

Download and extract the Node Exporter binary:

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz



$ tar -xzvf node_exporter-1.7.0.linux-amd64.tar.gz

Step 02:

Copy the Node Exporter binary to the appropriate location and set ownership:

$ sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Step 03:

Create a systemd unit file for Node Exporter:

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]

Description=Prometheus Node Exporter

Wants=network-online.target

After=network-online.target

[Service]

User=node_exporter

Group=node_exporter

Type=simple

ExecStart=/usr/local/bin/node_exporter

[Install]

WantedBy=multi-user.target

Start and enable the node_exporter service:

$ sudo systemctl daemon-reload



$ sudo systemctl enable --now node_exporter

You can retrieve the metrics served by Node Exporter like so:

$ curl localhost:9100/metrics

After setting up the Node Exporter, we add a new job under scrape_configs in the prometheus.yml file, specifying the private IP address and port (9100 by default) of the new Linux server. This setup enables Prometheus to scrape metrics from the Node Exporter running on the Linux server.

Step 04: Configure Prometheus to Scrape Metrics

Log in to your Prometheus server and configure Prometheus to scrape metrics from the new server.

$ vim /etc/prometheus/prometheus.yml

Locate the scrape_configs section and add a new entry under that section.

...

- job_name: 'Linux Server'

 static_configs:

 - targets: ['<PRIVATE_IP_ADDRESS_OF_NEW_SERVER>:9100']

...

Reload the Prometheus config:

$ sudo killall -HUP prometheus

Navigate to the Prometheus expression browser in your web browser using the public IP address of your Prometheus server: :9090.

Run some queries to retrieve metric data about your new server:

up



node_filesystem_avail_bytes

In summary, this lesson covers setting up a Prometheus exporter, configuring the Prometheus server to scrape metrics from this exporter, and verifying that the metrics are successfully being scraped. This process is crucial for monitoring and gathering insights about system performance and health.