If you’re running applications on Kubernetes using Amazon EKS and suddenly seeing logs appear in Datadog, you may have wondered:
“How did the logs even get there?”
Your application is running inside a Kubernetes pod.
Datadog is somewhere in the cloud.
Yet somehow every request, every error, and every stack trace magically appears in the Datadog UI.
At first, it feels invisible.
But once you understand the observability pipeline, Kubernetes starts making a lot more sense.
What Is Datadog?
Datadog is an observability platform.
It helps engineers monitor:
- Infrastructure
- Kubernetes clusters
- Applications
- Logs
- Metrics
- Traces
- Security events
Think of Datadog as a centralized monitoring brain for your systems.
Instead of SSH-ing into servers and manually checking logs, Datadog collects everything into one searchable place.
You can search:
service:map-service status:error
and instantly see logs from hundreds of Kubernetes pods.
You can also:
- Create dashboards
- Set alerts
- Trace requests
- Monitor pod restarts
- Watch CPU and memory usage
- Detect failures in real time
But here’s the important part:
Your application itself usually does NOT directly communicate with Datadog.
That job belongs to the Datadog Agent.
What Is the Datadog Agent?
The Datadog Agent is the collector.
It runs inside your Kubernetes cluster and continuously gathers:
- Logs
- Metrics
- Traces
- Kubernetes metadata
- Container information
In Kubernetes, the Datadog Agent is usually deployed as a:
DaemonSet
A DaemonSet means:
“Run one Datadog Agent pod on every Kubernetes node.”
So if your EKS cluster has:
20 worker nodes
Kubernetes automatically creates:
20 Datadog Agent pods
Each agent watches workloads running on its own node.
The Big Question
Here’s what most people wonder:
“My application is inside a pod… so how does Datadog see its logs?”
To understand this, we first need to understand how Kubernetes handles logs internally.
Step 1 Your Application Writes Logs
Inside your container, your application usually writes logs to:
stdout
stderr
For example:
print("hello world")
or:
logger.error("database connection failed")
Or maybe your Gunicorn server prints:
500 Internal Server Error
Your app is simply writing text output.
It doesn’t know anything about Datadog.
It doesn’t know about dashboards.
It doesn’t know about observability.
It is simply talking.
Step 2 Kubernetes Captures Container Logs
Kubernetes containers run through a container runtime like:
- containerd
- Docker
- CRI-O
In EKS today, most clusters use:
containerd
The runtime captures container stdout/stderr and stores it as log files on the Kubernetes node.
Usually under paths like:
/var/log/containers/
/var/log/pods/
At this point:
The logs now physically exist on the EC2 worker node.
This is the key realization:
Your pod logs are not floating magically inside Kubernetes.
They become actual files on the node filesystem.
Step 3 The Datadog Agent Watches Those Logs
Now the Datadog Agent enters the picture.
Because the Agent runs on every node, it can monitor container log files locally.
Conceptually, it does something similar to:
tail -f /var/log/containers/*.log
The Agent continuously watches:
- new logs
- new containers
- restarted pods
- Kubernetes metadata
Whenever your application writes a log like:
ERROR database timeout
the Datadog Agent immediately sees it.
Step 4 Metadata Enrichment
This is where Datadog becomes powerful.
The Datadog Agent doesn’t just forward raw text.
It enriches logs with Kubernetes metadata.
Example:
{
"message": "GET /orders 500",
"pod_name": "map-service-12345abc-west",
"namespace": "production",
"service": "map-service",
"node": "ip-10-0-10-12",
"cluster": "eks-prod"
}
Now your logs become searchable.
You can filter by:
- pod
- namespace
- service
- cluster
- environment
- container
Without this enrichment, logs would just be random text.
Step 5 Secure Upload to Datadog Cloud
After enrichment, the Agent securely uploads logs to Datadog’s backend using HTTPS.
The flow now looks like this:
That’s the full hidden journey.
What About Metrics?
Datadog does more than logs.
The Agent also collects metrics from Kubernetes.
For example:
- CPU usage
- Memory usage
- Pod restarts
- RabbitMQ queue depth
- Redis connections
- Network traffic
It gathers metrics from:
- kubelet
- Kubernetes API
- cAdvisor
- integrations
- DogStatsD
Example:
Kubernetes Node
↓
kubelet metrics
↓
Datadog Agent
↓
Datadog Cloud
This is why you can build dashboards showing:
- pod CPU %
- node memory
- HPA scaling
- container restarts
in real time.
What About Traces (APM)?
Datadog APM tracks requests flowing across services.
Example:
Frontend
↓
API Gateway
↓
ROS Service
↓
PostgreSQL
Datadog can measure:
- request latency
- database queries
- failed API calls
- slow endpoints
This works using tracing libraries like:
- ddtrace (Python)
- datadog-js
- Java agent
These libraries send trace data to the Datadog Agent.
Then the Agent forwards it to Datadog.
Why Does the Datadog Agent Need Permissions?
This part surprises many engineers.
The Datadog Agent often mounts host system paths like:
/var/log/containers
/proc
/sys/fs/cgroup
Why?
Because Kubernetes isolation prevents containers from normally seeing host resources.
The Agent needs visibility into:
- container logs
- processes
- cgroups
- node metrics
Some advanced integrations require even more access.
For example, the Gunicorn integration may require:
hostPID: true
privileged: true
SYS_PTRACE
because the Agent sometimes needs to inspect processes outside its own container namespace.
This is why observability agents can become security discussions between:
- platform teams
- DevOps engineers
- security teams
Monitoring deeply often requires elevated visibility.
The Biggest Realization
Your application is usually just writing logs normally.
The Datadog Agent acts like a collector sitting beside your workloads.
A simple analogy:
Application = person speaking
Datadog Agent = microphone
Datadog Cloud = recording studio
Datadog UI = playback/search system
The application just talks.
The Agent listens.
Once you understand this pipeline, debugging Kubernetes becomes much easier.
When logs disappear, you know exactly where to investigate:
- Is the app writing logs?
- Is stdout working?
- Did the container runtime capture them?
- Is the Datadog Agent healthy?
- Are hostPath mounts correct?
- Is networking blocking uploads?
- Is metadata enrichment failing?
Observability suddenly becomes understandable instead of magical.
And honestly, that’s one of the coolest parts of Kubernetes infrastructure.
Behind every “simple dashboard” is an entire hidden pipeline quietly moving data across your cluster 24/7.
SOCIAL SHARE CARD GENERATOR