According to the is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of pull. It is designed to be very cost-effective and highly scalable. Unlike other logging systems, Loki does not index the contents of the logs but only indexes metadata about your logs as a set of labels for each log stream.
A log stream is a set of logs that share the same labels. Labels help Loki to find a log stream within your data store, so having a quality set of labels is key to efficient query execution.
Log data is then compressed and stored in chunks in an object store such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS) or, for development or proof of concept, on the file system. A small index and highly compressed chunks simplify the operation and significantly lower Loki's cost. Now, we can understand the Loki deployment modes.
Loki Deployment modes
Loki is a distributed system composed of multiple microservices, each responsible for specific tasks. These microservices can be deployed independently or together in a unique build mode where all services coexist within the same binary. Understanding the available deployment modes helps you decide how to structure these microservices to achieve optimal performance, scalability, and resilience in your environment. Different modes will impact how Loki's components—like the Distributor, Ingester, Querier, and others—interact and how efficiently they manage logs.
The list of Loki microservices includes:
- Cache Generation Loader
- Compactor
- Distributor
- Index-gateway
- Ingester
- Ingester-Querier
- Overrides Exporter
- Querier
- Query-frontend
- Query-scheduler
- Ruler
- Table Manager (deprecated)
Different deployment modes
Loki offers different deployment modes, which allow us to build a highly available logging system. We need to choose the modes considering our log reads/writes rate, maintenance overhead, and complexity. Loki can be deployed in three modes, each suited for varying scales and complexity.
Monolithic mode: The is the preferred mode for most installations and is the default configuration when installing Loki via Helm charts. This mode balances simplicity and scalability by separating the execution paths into distinct targets: READ, WRITE, and BACKEND. These targets can be scaled independently based on business needs, allowing this deployment to handle up to a few terabytes of logs per day. The SSD mode requires a reverse proxy, such as Nginx, to route client API requests to the appropriate read or write nodes, and this setup is included by default in the Loki Helm chart.
Microservices Deployment mode: The code snippet to create the required AWS resources, such as IAM roles, policies, and an S3 bucket with appropriate bucket policies. This code helps automate the provisioning of the necessary infrastructure, ensuring that you have a consistent and repeatable environment for running Loki with high availability.
Guide to install Loki
Following the guide, you can
In this setup, we are running multiple replicas of Loki read, write, and backend pods.
With a replication_factor of 3, it is imperative to ensure that both the write and backend are operating with 3 replicas; otherwise, the quorum will fail, and Loki will be unavailable.
The following image illustrates Loki's integration with Amazon S3 for log storage in a single-tenant environment. In this configuration, logs are organized into two primary folders within the S3 bucket: index and fake.
Index folder: This folder contains the index files that allow Loki to efficiently query and retrieve log data. The index serves as a mapping of log entries, enabling fast search operations and optimizing the performance of log retrieval.
Fake folder: This folder is used to store the actual log data. In a single-tenant setup, it may be labeled as "fake," but it holds the important logs generated by your applications.
, we should also be able to verify the logs by querying against Loki instances.
Exploring approaches for disaster recovery
Loki is a critical component of our application stack, responsible for aggregating logs from multiple microservices and displaying them in the web application console for end-user access. These logs need to be retained for an extended period—up to 90 days.
As part of our disaster recovery (DR) strategy for the application stack, ensuring the availability and accessibility of logs during a disaster is crucial. If Region-1 becomes unavailable, the applications must continue to run and access logs seamlessly. To address this, we decided to implement high availability for Loki by running two instances in separate regions. If one Loki instance fails, the instance in the other region should continue to handle both read and write operations for the logs.
We explored three different approaches to setting up DR for Loki, intending to enable read and write capabilities across both Region-1 and Region-2, ensuring fault tolerance and uninterrupted log management.
Approach 1: Implementing S3 Cross-Region Replication
.
Flushing logs and configuration parameters
When logs are generated, Loki stores them in chunks, which are temporary data structures that hold log entries before they are flushed to the object store (in this case, S3). The flushing process is controlled by two critical parameters: max_chunk_age and chunk_idle_period.
:
The
chunk_idle_periodparameter determines how long Loki waits for new log entries in a stream before considering that stream idle and flushing the chunk. A lower value (less than 2 hours) can lead to the creation of too many small chunks, increasing the storage I/O demands.On the other hand, setting a higher value (greater than 2 hours) allows inactive streams to retain logs in memory longer, which can enhance retention but may lead to potential memory inefficiency if many streams become idle.
This example shows querying logs from one Loki instance, which is pointed to CRR-enabled S3 bucket.
With this approach, in the event of a disaster or failover in one region, there is a risk of losing up to 2 hours of log data. This potential data loss occurs because logs that have not yet been flushed from memory to the S3 bucket during that time frame may not be recoverable if the ingester fails.
Also, . Several factors affect replication time, including:
- The size of the objects to replicate.
- The number of objects to replicate.
For example, if Amazon S3 is replicating more than 3,500 objects per second, then there might be latency while the destination bucket (MRAP) offer a global endpoint for routing S3 request traffic across multiple AWS Regions, simplifying the architecture by eliminating complex networking setups. While Loki does not directly support MRAP endpoints, this feature can still enhance your logging solution. MRAP allows for centralized log management, improving performance by routing requests to the nearest S3 bucket, which reduces latency. It also boosts redundancy and reliability by rerouting traffic during regional outages, ensuring logs remain accessible. Additionally, MRAP can help minimize cross-region data transfer fees, making it a cost-effective option. However, at the time of this writing, there is a known bug that prevents Loki from effectively using this endpoint. Understanding MRAP can still be beneficial for future scalability and efficiency in your logging infrastructure.
, a lightweight and ultra-fast tool for building observability pipelines. With Vector, we could collect, transform, and route logs to AWS S3.
- So, our infrastructure is one S3 bucket and Loki per region.
- Vector will be running as a sidecar with the application pods.
- Since EKS clusters are connected via a transit gateway, we configured a private endpoint for both the Loki instances. We don't want to expose it to the public as it contains application logs.
- Configured vector sources to read the application logs, transform and sink, and write to both the Loki instance.
collect or receive data from observability data sources into Vector.
send data onward from Vector to external services or destinations.
data_dir: /vector-data-dir
sinks:
# Write events to Loki in the same cluster
loki_write:
encoding:
codec: json
endpoint: http://loki-write.loki:3100
inputs:
- my_transform_id
type: loki
# Write events to Loki in the cross-region cluster
loki_cross:
encoding:
codec: json
endpoint: https://loki-write.aws-us-west-2.loki
inputs:
- my_transform_id
type: loki
# Define the source to read log file
sources:
my_source_id:
type: file
include:
- /var/log/**/*.log
# Define the transform to parse syslog messages
transforms:
my_transform_id:
type: remap
inputs:
- my_source_id
source: . = parse_json(.message)
In this setup, Vector collects logs from the /var/log/ directory and internal Vector logs.
It parses as JSON, and replaces the entire event with the parsed JSON object and sends them to two Loki destinations (local and cross-region). The configuration ensures logs are sent in JSON format and can handle errors during log processing.
Conclusion
The journey to achieving high availability (HA) and disaster recovery (DR) for Loki has been challenging and enlightening. Through exploring various deployment modes and approaches, we've gained a deeper understanding of ensuring our logging system can withstand and recover from potential disruptions. The successful implementation of a Simple Scalable Mode with an S3 backend and the innovative use of Vector as a sidecar have fortified our system's resilience and underscored the importance of proactive planning and continuous improvement in our infrastructure.
I hope you found this post informative and engaging. I’d love to hear your thoughts on this post; let’s connect and start a conversation on .
SOCIAL SHARE CARD GENERATOR