Trace Sampling for LLM Apps: Keep the Spans That Matter, Drop the Rest

processors:

  tail_sampling:

    decision_wait: 10s

    num_traces: 100000

    policies:

      - name: keep-errors

        type: status_code

        status_code:

          status_codes: [ERROR]



      - name: keep-slow

        type: latency

        latency:

          threshold_ms: 10000



      - name: keep-expensive

        type: numeric_attribute

        numeric_attribute:

          key: gen_ai.usage.cost_usd

          min_value: 1



      - name: keep-eval-traffic

        type: string_attribute

        string_attribute:

          key: eval.tag

          values: [canary, regression, review]



      - name: sample-the-rest

        type: probabilistic

        probabilistic:

          sampling_percentage: 5

A few things to know about this processor. decision_wait is how long the Collector holds a trace's spans after the last one arrives; set it longer than your slowest expected trace or you will make decisions on incomplete data. num_traces is the in-memory buffer size, and it is a memory cost you have to budget for. The policies are an OR: a trace is kept if it matches any policy, so the cheap probabilistic rule never overrides a keep.

The cost attribute (gen_ai.usage.cost_usd) is not standard. You set it yourself at instrumentation time, computed from the token counts the provider returns. The point is that tail sampling can route on any attribute you put on the span, so put the ones you want to filter on there.

Setting the attributes the policy reads

Tail rules are only as good as the span attributes you feed them. At instrumentation time, stamp the trace with what the Collector will need to decide.

from opentelemetry import trace



tracer = trace.get_tracer("llm-app")



# illustrative rates — set from your provider's pricing

PRICE_PER_1K = {"input": 0.003, "output": 0.015}



def record_llm_call(model, prompt, response, usage, tag=None):

    with tracer.start_as_current_span("llm.chat") as span:

        span.set_attribute("gen_ai.request.model", model)

        in_tok = usage["input_tokens"]

        out_tok = usage["output_tokens"]

        cost = (

            in_tok / 1000 * PRICE_PER_1K["input"]

            + out_tok / 1000 * PRICE_PER_1K["output"]

        )

        span.set_attribute("gen_ai.usage.cost_usd", cost)

        span.set_attribute("gen_ai.usage.input_tokens", in_tok)

        span.set_attribute("gen_ai.usage.output_tokens", out_tok)

        if tag:

            span.set_attribute("eval.tag", tag)

        return response

When you run your hourly canary or a regression suite, set tag="canary" on those calls. The keep-eval-traffic policy then pins them at 100%, so your offline comparisons always have the full trace, never a sampled gap. A canary you only kept 5% of the time is a canary you cannot trust.

Two traps worth naming

The probabilistic rule undercounts your real volume. Once you sample the healthy traffic at 5%, any metric you compute from stored traces — request count, average cost, token throughput — is off by the sampling rate unless you correct for it. The fix is to derive volume and cost metrics from a separate, unsampled metrics pipeline, and treat traces as exemplars, not as the source of truth for counts. Sample your traces; never sample your counters.

Tail sampling does not compose with load balancing for free. The tail_sampling processor needs every span of a trace to land in the same Collector instance, because it decides per trace ID. If your spans fan out across a pool of Collectors behind a round-robin load balancer, a trace gets split and the decision is made on a fragment. The standard fix is a two-tier setup: a first tier that routes by trace ID (the