Tracking LLM Latency & Cost: How I Instrumented an AI Agent Pipeline Using OpenTelemetry and SigNoz

The Problem: The High Cost of Blind AI Requests
When you build a standard CRUD application, performance bottlenecks are predictable: it's almost always a missing database index or a heavy payload. With AI agents and LLMs, performance is wildly unpredictable.

A single user prompt can trigger multiple API calls, prompt formatting, vector database lookups (RAG), and streaming responses. If a user complains that the app is slow, you can't just check your server CPU usage. You need to know:

Which specific LLM call took the longest?

Did our vector database search slow down the context retrieval?

How many tokens did that request consume (so we don't go broke)?

To answer this, I instrumented a Python-based AI agent backend using OpenTelemetry and hooked it up to SigNoz. Here is exactly how to do it.

Show Your Work: The Setup & Instrumentation
Instead of relying on generic auto-instrumentation that only tells you an HTTP request happened, we are going to write explicit, semantic inline tracing. This gives us deep, domain-specific insights like token counts and model names right inside our telemetry.

1. The OpenTelemetry Initialization Script First, we configure our tracer provider to send data directly to our SigNoz collector. Create a file named tracing.py:

Python
`
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def init_tracer(service_name="ai-agent-service"):
# Define application metadata
resource = Resource.create(attributes={"service.name": service_name})
provider = TracerProvider(resource=resource)

# Configure the OTLP exporter to point to SigNoz (Default gRPC port is 4317)
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)

# Add the batch processor to handle spans efficiently without blocking application logic
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)

trace.set_tracer_provider(provider)
return trace.get_tracer(service_name)

tracer = init_tracer()

1. Wrapping the AI Agent Logic Now, let’s look at the core application code. We use the custom tracer to create a parent span for the complete agent execution, and individual nested child spans for the database lookup and the actual external AI API call.

Python
import time
from tracing import tracer
from opentelemetry.trace import StatusCode

def fake_vector_db_search(query):
with tracer.start_as_current_span("vector_db_search") as span:
span.set_attribute("db.system", "chromadb")
span.set_attribute("db.query", query)

    # Simulating vector embedding search latency
    time.sleep(0.4) 
    return "Context: OpenTelemetry is an open-source observability framework."

def call_llm_agent(user_prompt):
# Start the top-level parent trace
with tracer.start_as_current_span("ai_agent_pipeline") as parent_span:
parent_span.set_attribute("user.prompt_length", len(user_prompt))

    # Step 1: Retrieve Context (Child Span 1)
    context = fake_vector_db_search(user_prompt)

    # Step 2: Execute LLM Call (Child Span 2)
    with tracer.start_as_current_span("llm_generation") as llm_span:
        # Semantic attributes help us filter data in SigNoz later
        llm_span.set_attribute("llm.model_name", "gpt-4o")
        llm_span.set_attribute("llm.temperature", 0.7)

        try:
            # Simulating external AI API latency
            start_time = time.time()
            time.sleep(1.8) 
            generation_time = time.time() - start_time

            # Mocking a successful API response payload
            prompt_tokens = 120
            completion_tokens = 250

            # Capture critical domain/cost metrics
            llm_span.set_attribute("llm.prompt_tokens", prompt_tokens)
            llm_span.set_attribute("llm.completion_tokens", completion_tokens)
            llm_span.set_attribute("llm.total_tokens", prompt_tokens + completion_tokens)
            llm_span.set_attribute("llm.latency_seconds", generation_time)

            llm_span.set_status(StatusCode.OK)
            return "AI Response: Steps completed successfully."

        except Exception as e:
            llm_span.record_exception(e)
            llm_span.set_status(StatusCode.ERROR, description=str(e))
            raise e

if name == "main":
print("Running instrumented AI Agent...")
call_llm_agent("How do I properly monitor my pipeline?")
`
Tracking the Output in SigNoz
Once you spin up SigNoz via Docker and run the script above, the exact execution flow transfers from your terminal into a visual powerhouse.

Analyzing the Flame Graph
When you open the Traces tab in SigNoz and click into ai_agent_pipeline, you instantly see the breakdown of the 2.2-second request:

You see the total ai_agent_pipeline bar spans across the entire duration.

Directly underneath it, you see vector_db_search taking exactly 400ms.

Immediately following that, llm_generation takes up 1.8 seconds of the timeline.

If a request spikes to 5 seconds, you no longer have to guess. The flame graph visually exposes exactly which child span shifted right.

Creating a Custom Token & Cost Dashboard
Because we explicitly attached attributes like llm.total_tokens and llm.model_name directly to our spans, we don't just get charts for network speeds—we can build business-critical dashboards.

In SigNoz, you can build a custom panel using ClickHouse queries on your span attributes:

SQL
SELECT
sum(attributes_number_value[indexOf(attributes_string_key, 'llm.total_tokens')]) as total_tokens_consumed,
attributes_string_value[indexOf(attributes_string_key, 'llm.model_name')] as model
FROM signoz_traces.spans
WHERE attributes_string_key = 'llm.model_name'
GROUP BY model

This transforms raw engineering trace telemetry into an executive-level dashboard showing real-time token spend per AI model.

Key Gotchas & What I Learned
Writing this implementation exposed a few critical engineering details that the generic setup guides completely skip over:

📌 Don't Over-Trace Async Streams: If you are streaming responses token-by-token (using stream=True in OpenAI/LangChain), wrapping the entire loop will artifically inflate your generation latency metrics because it includes the client's network read speed. Instead, explicitly measure the time from the initial request to the first byte received to calculate accurate Time-To-First-Token (TTFT).

📌 Mind the Batch Processor: Always use BatchSpanProcessor in staging and production environments. Using a synchronous SimpleSpanProcessor will force your backend application to wait for the SigNoz collector to acknowledge receipt of data before responding to your user, completely destroying application performance.

Conclusion
Observability shouldn't stop at checking if your server container is healthy. By mapping OpenTelemetry custom spans directly to our AI pipelines, we can isolate external vendor API slowdowns, track precise token resource consumption, and debug complex agent architectures with zero guesswork.

To learn more about OpenTelemetry semantic conventions for AI architectures, check out the Official OpenTelemetry Documentation.

To deploy your own observability stack locally, clone the SigNoz GitHub Repository.