If you’ve ever felt overwhelmed by words like “collector,” “instrumentation,” or “tracing,” you’re not alone. I’m right there with you 😵💫, still learning and piecing everything together. But as I keep connecting the dots, tools like OpenTelemetry are slowly starting to make more sense. Attending (OTel) as a toolset for understanding what’s happening inside your software. Imagine you are organizing a large mailing process where you need to send packages to people. You want to know:
When each package 📦 is picked up.
Where it goes next ❓.
When it gets delivered .
Let's break down some of the components of OpenTelemetry using this analogy:
Instrumentation: Attach a tracking label to each package. This is how you collect data about each package's journey. In software, this means adding sensors to your code to collect data about what’s happening.
Logs: Logs are like notes that track every event happening during the package's journey, like when it enters a facility or leaves for delivery. You can export logs to different backends, such as a logging service, to keep track of everything.
Metrics: Metrics are like summaries that tell you how many packages were delivered in a day, how long deliveries took, or how many were delayed. These metrics can be exported to specialized monitoring services to analyze overall performance.
Traces: Traces are like a detailed route map showing the complete journey of an individual package, from pickup to delivery. They can be exported to a different observability platform that helps you analyze each step of the journey in detail.
Collector: Imagine a central hub where all tracking information is gathered. The Collector collects data from all the tracking labels, processes it, and sends it for analysis. The Collector can gather data from multiple sources and send it to different places. However, it isn't always necessary, you can export data directly from your app to an observability platform.
Receiver: The Collector uses receivers to gather data from different sources. A receiver is like the entry point for data into the Collector, allowing it to receive telemetry data from various components like your application, services, or other systems.
Processor: Once the data is received, the processor takes over. Think of the processor as the person in the central hub who organizes, filters, and enhances the tracking information before it is sent out. Processors allow you to modify the data, such as adding additional attributes or dropping unnecessary information, ensuring that only the most relevant data moves forward.
Exporter: The Exporter sends the collected data to its final destination, like an observability platform or database. You can configure your app to export directly to a backend or use a Collector to handle the exporting.
Observability Platform: This is like a dashboard that shows the progress of all packages. It turns the collected data into insights, graphs, alerts, and dashboards, so you can understand what's happening.
Using OpenTelemetry
Currently, I am working with:
- Azure Kubernetes Service (AKS)
- Database: For data storage.
- Redis: For caching.
- Azure App Configuration: To manage application settings centrally.
- Java with SpringBoot
We need a way to track the interactions and flow between all these components. This will help us understand how they work together and gain valuable insights, not just identify potential issues.
Starting with Azure Monitor
My journey began with the Azure Monitor instrumentation library in my Spring Boot app. This approach allowed me to
Limitations
I wanted to add custom properties to the telemetry data and be more intentional with the data we are sending, ensuring we collect only what is necessary and meaningful.
Code Changes:
Adding custom properties was not as straightforward as I hoped. (Note: There might be a way to do this without modifying code, but I couldn’t get it to work.). For more intentional insights tailored to our business needs, manual instrumentation is required.
Vendor Lock-In: I also wanted to avoid being locked into a specific vendor.
Experimentation: I had been experimenting with OpenTelemetry on the side, specifically using auto-instrumentation for Python on one of my FastAPI services.
Switching to OpenTelemetry
Initial Goal: I wanted to avoid using a Collector and instead export telemetry data directly to Azure. The Collector adds an extra layer of setup and configuration, which feels overwhelming while I'm still trying to grasp all of this.
Limitation: The could be used to send data directly to Azure and avoid the Collector while using the auto-instrumentation Spring starter for OpenTelemetry.
Introducing the Collector: Although my initial goal was to avoid adding a to deploy the Collector in our AKS cluster.
Collector Contrib Image: I referenced the Collector contrib image, which includes Azure Monitor as an available exporter.
Outcome: Since we have a lot of services, the Collector gives us the ability to transform and enrich data before exporting it. Additionally, having a central place for processing data makes managing and scaling telemetry across multiple services much easier.
Setup without Collector
+----------------------------------------+
| Instrumented Applications |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
| logs) using OTel SDK |
| - Has its own Exporter 🚀 |
+-----------------|----------------------+
|
v
+----------------------------------------+
| 🛰️ Observability Backend |
|----------------------------------------|
| - Data Storage |
| - Query & Visualization (e.g.,Azure |
| Honeycomb, Grafana, Prometheus |
| 📊 Insights and Monitoring |
+----------------------------------------+
Setup with Collector
+----------------------------------------+
| Instrumented Applications |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
| logs) using OTel SDK |
| - Has simple Exporter to Collector 🚀 |
+-----------------|----------------------+
|
v
+----------------------------------------+
| 🚀 OpenTelemetry Collector |
|----------------------------------------|
| 📥 Receivers |
| - Collect telemetry data from apps |
| |
| 🔄 Processors |
| - Transform and enrich data |
| |
| 📤 Exporters |
| - Send data to observability backend |
+-----------------|----------------------+
|
v
+----------------------------------------+
| 🛰️ Observability Backend |
|----------------------------------------|
| - Data Storage |
| - Query & Visualization (e.g., Azure |
| Honeycomb, Prometheus, Grafana) |
| 📊 Insights and Monitoring |
+----------------------------------------+
Sample values.yaml for deploying collector to AKS using Helm
image:
repository: "otel/opentelemetry-collector-contrib"
tag: "latest"
config:
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
send_batch_size: 1024
exporters:
azuremonitor:
connection_string: "<your-connection-string>"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [azuremonitor]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [azuremonitor]
logs:
receivers: [otlp]
processors: [batch]
exporters: [azuremonitor]
extensions:
health_check:
endpoint: "0.0.0.0:13133"
SOCIAL SHARE CARD GENERATOR