A request enters your system through an API gateway, hits an authentication service, queries a database, calls a payment provider, publishes an event to a message queue, and returns a response. When that request takes 4 seconds instead of 400 milliseconds, which service is responsible?
Without distributed tracing, you open five dashboards, compare timestamps in five different log streams, and try to reconstruct the request path from memory. With distributed tracing, you open one trace and see every hop, every duration, and every failure — in a single view.
Distributed tracing is the practice of propagating a unique identifier through every service that handles a request, recording the work each service does as spans, and assembling those spans into a trace that represents the request's complete journey.
The mental model: spans and traces
A span is a named, timed operation. "Query user table" is a span. "Call Stripe API" is a span. "Validate JWT" is a span. Each span records:
- A name (what happened)
- A start time and duration (how long it took)
- A status (OK, error, or unset)
Attributes (key-value metadata:http.method=POST,db.statement=SELECT...,rpc.service=PaymentService)- A parent span ID (which span triggered this one)
A trace is a tree of spans rooted at the entry point. The root span represents the entire request. Child spans represent sub-operations. The parent-child relationships form a directed acyclic graph that mirrors the actual execution flow.
Trace: a]b2c3d4 (POST /api/v1/orders)
├── [12ms] Validate JWT
├── [340ms] Query order history
│ └── [320ms] PostgreSQL SELECT
├── [1,200ms] Call Stripe API
│ ├── [800ms] Create PaymentIntent
│ └── [380ms] Confirm PaymentIntent
└── [45ms] Publish OrderCreated event
└── [38ms] NATS publish
From this trace, you can immediately see that the Stripe API call dominates the latency (1,200ms out of ~1,600ms total). No log correlation, no dashboard cross-referencing, no guesswork.
Context propagation: the glue
Spans only form a trace if each service knows which trace it's participating in. This happens through context propagation — injecting the trace ID and parent span ID into the request headers, then extracting them on the receiving side.
The standard header format is
SOCIAL SHARE CARD GENERATOR