🔧 Centralize and visualize Docker logs in Grafana with Loki
Nachrichtenbereich: 🔧 Programmierung
🔗 Quelle: dev.to
In this article:
- Introduction
- How does it work?
- 1. Setting up Loki
- 2. Setting up the Docker Loki Docker plugin
- 3 Grafana!
- Conclusion
Introduction
I've previously written about how to centralize and visualize logs in Grafana with Loki for apps that write logs to files here.
Doing the same with Docker is SO MUCH SIMPLER.
My approach
You will see other guides showing you how to set up everything in a single machine. While this might be convenient for people running all their Docker stuff in a single server, it's not the best practice for a production environment.
I will show you how to set up everything in a way that you can scale horizontally. With Grafana, Loki and Promtail each running in isolation of each other.
I.e.: You can have promtail in a server running Docker containers for a web app, sending the logs to a Loki instance running in another server and then visualize everything in Grafana running in a third server.
How does it work?
The set up is really similar to the one I wrote about in the previous article, but with a few differences:
The main difference is that, with this setup, we have the Docker daemon sending the logs directly to Loki, without having to configure Promtail at all!
This has a few advantages:
- You don't need to mess around with log files and their locations
- Docker takes care of sending the logs to Loki
- Querying the logs becomes SO MUCH EASIER in Grafana as we can query by container name, image, compose project, etc.
1. Setting up Loki
If you already have a Loki instance and are only interested in setting up the Docker Loki plugin, you can skip this section.
As I said in the introduction, I like to set up everything in separate servers, so here's how I set up Loki in a Ubuntu server in my home lab.
Note that this setup also works if you want to do it all in one machine, it's just that you might want to set up loki and grafana in the same compose file.
1.1 Create a directory for our configuration files
cd && mkdir loki && cd loki
1.2 Create a loki-config.yaml
file
nano loki-config.yaml
Paste the following content:
# You can enable authentication to prevent unauthorized access to your logs
auth_enabled: false
server:
# This is the port where Loki will receive logs from Docker (or Promtail)
http_listen_port: 3100
# This is the port where Loki will expose its API
grpc_listen_port: 9096
common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
1.3 Create a docker-compose.yaml
file
nano docker-compose.yaml
Paste the following content:
services:
loki:
image: grafana/loki:latest
volumes:
- ./loki-config.yml:/etc/loki/loki-config.yml
ports:
- "3100:3100"
restart: unless-stopped
command: -config.file=/etc/loki/loki-config.yml
Even though you could run this single container with
docker run
, I like to usedocker-compose
so I don't have to remember the command to start the container.
1.4 Start Loki!
docker compose up -d
If everything went well, you should be able to access Loki at http://your-server-ip:3100
.
If you have servers in multiple locations (aws, digital ocean, home lab, etc), you can create a reverse proxy with NGINX or a similar service to expose Loki to the internet. If so, I recommend either enabling authentication in the loki-config.yaml
file or allowing only your servers to access Loki via the reverse proxy or firewall.
Now you can point all your Docker servers to point to this Loki instance!
2. Setting up the Docker Loki Docker plugin
I'll assume you already have one or multiple Docker containers running in a server somewhere.
You can repeat this step for every server you have running Docker containers.
2.1 Create the necessary files
What I like to do in my servers is to set up a directory only for this logging setup, so:
cd && mkdir promtail && cd promtail
In this directory you will need 2 files:
promtail-config.yaml
docker-compose.yaml
2.2 promtail-config.yaml
This file will tell Promtail where to look for the logs and where to send them.
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
# Temporary file to store the last read position for each log stream. Useful in case of a crash or restart.
filename: /tmp/positions.yaml
clients:
# The URL where your Loki instance is running
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: docker
pipeline_stages:
- docker: {}
static_configs:
- labels:
job: docker
__path__: /var/lib/docker/containers/*/*-json.log
You can directly copy the code above, the only important thing to change is the url
field in the clients
section.
The code above assumes you will run a Docker container running Loki in the same server as Promtail (thus the localhost
). I, for instance, have Loki in my home lab running behind a reverse proxy, so I would change this to https://myloki.mydomain.com/loki/api/v1/push
.
2.3 docker-compose.yaml
# docker-compose.yml
services:
promtail:
image: grafana/promtail:latest
restart: unless-stopped
volumes:
- /var/log:/var/log
- ./promtail-config.yml:/etc/promtail/promtail-config.yml
command: -config.file=/etc/promtail/promtail-config.yml
This docker-compose file will start a Promtail container with the configuration file we just created.
2.4 Install the Docker loki plugin
This is the easiest part.
docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions
Check here for the latest version of this command.
If the command was successful, you should see the plugin listed when you run docker plugin ls
.
2.5 Configure the Docker daemon
We need to create a daemon.json
file in /etc/docker/
with the following content:
{
"log-driver": "loki",
"log-opts": {
"loki-url": "http://localhost:3100/loki/api/v1/push",
"loki-batch-size": "400"
}
}
This tells Docker that it should use the loki log driver instead of the default one and sends the logs to the Loki instance.
Again, change the
loki-url
field to the correct URL for you.
loki-batch-size
is optional, but I like to set it to 400 to avoid sending too many requests to Loki.
2.6 Wait, if the Loki plugin sends the logs, why do we need Promtail?
The Loki plugin will send the logs to Loki, but it won't keep track of the log position. This means that if Loki goes down, you will lose the logs that were sent while it was down.
Promtail will keep track of the log position and send the logs to Loki when it's back up.
2.7 Restart Docker
sudo systemctl restart docker
2.8 Start promtail!
cd ~/promtail && docker compose up -d
Great! Now we have all the Docker containers in this machine sending their logs to our Loki instance!
3 Grafana!
This is the final and coolest part!
3.1 Install Grafana
If you already have Grafana set up, you can skip ahead to 3.3 where I show how easy is to query the logs.
It's as simple as running a container:
docker run -d --name=grafana -p 3000:3000 grafana/grafana
You can access Grafana at http://your-server-ip:3000
.
3.2 Connect Grafana to Loki
In the left panel, go to "Connections" > "Data Sources".
Click on "+ Add new data source".
Search for "Loki":
- Fill in a name and the URL of your Loki instance:
- Click on "Save & Test". You should see a message saying everything is working correctly.
3.3 Query the logs
In the left panel, go to "Explore".
Select your Loki source at the top:
- 🎉 Now comes the super cool part of this setup. You can query the logs by container name, compose project, etc.:
This is the best part of this setup, as you can really easily see the logs of a specific container without having to sort through hundreds of files and weird configurations.
Conclusion
I love this setup, I deploy my stuff mainly with Docker and having all the logs centralized in a single place makes my life so much easier to debug and monitor.
Hope you found this guide useful! If you have any questions or suggestions, feel free to reach out to me on Twitter!
Also, if you want to learn more about how to query Loki logs in Grafana, check out this article.
...
🔧 Pushing Cloudflare Worker logs to Grafana Loki
📈 41.47 Punkte
🔧 Programmierung
🔧 Grafana Loki and Kubernetes Event exporter
📈 31.33 Punkte
🔧 Programmierung
🕵️ Low CVE-2019-15635: Grafana Grafana
📈 26.22 Punkte
🕵️ Sicherheitslücken
🕵️ Low CVE-2020-12052: Grafana Grafana
📈 26.22 Punkte
🕵️ Sicherheitslücken
🕵️ Low CVE-2020-13430: Grafana Grafana
📈 26.22 Punkte
🕵️ Sicherheitslücken
🕵️ Low CVE-2018-18624: Grafana Grafana
📈 26.22 Punkte
🕵️ Sicherheitslücken