LLM Observability with OpenTelemetry

Your LLM feature works in the demo. Then it ships. A week later finance forwards a cloud bill with a line item that tripled, a customer says the “summarize” button “sometimes just spins forever,” and your only evidence is a log line that says POST /chat 200. You cannot tell which prompts are expensive, which model version answered, or where the three seconds went.

Here is the uncomfortable truth: an LLM call is a network call to a slow, non-deterministic, metered black box, and you are treating it like a local function. You cannot debug or cost-control what you cannot see. OpenTelemetry (OTel) fixes this by giving every call a span, a token count, and a latency number, using the same tracing pipeline you already run for the rest of your backend. This is the same discipline covered in logs, traces, and metrics with OpenTelemetry, pointed at the part of your stack that bleeds money.

Why LLM calls need tracing more than your CRUD endpoints

A normal HTTP handler fails loudly: a 500, a stack trace, a timeout you can reproduce. LLM calls fail in ways your metrics dashboard was never designed to catch:

• Latency is bimodal. A p50 of 800ms hides a p99 of 30 seconds when the model streams a long completion or the provider is degraded.
• Cost is per-token and invisible. A one-line prompt change that adds a 4,000-token system context does not throw an error. It just quietly doubles your bill.
• Failures are semantic, not HTTP. The API returns 200 OK with finish_reason: "length", meaning the model was truncated mid-answer. Your monitoring sees success.
• The inputs drift. As the prompt is not the product argues, the prompt and the model version are part of your runtime. If you cannot see which version produced a bad answer, you cannot regress-test it.

Tracing turns each of those into a queryable attribute.

The GenAI semantic conventions

OpenTelemetry publishes semantic conventions so that “which model” and “how many tokens” mean the same thing across OpenAI, Anthropic, Bedrock, and your own gateway. The GenAI attributes live under the gen_ai.* namespace. These are the ones you will set most often (verified against the current registry, which recently renamed several of them):

Attribute	Meaning
gen_ai.provider.name	Provider, e.g. openai, anthropic, aws.bedrock (replaces the old gen_ai.system)
gen_ai.operation.name	The operation, e.g. chat, embeddings
gen_ai.request.model	Model requested, e.g. gpt-4o
gen_ai.response.model	Model that actually answered
gen_ai.usage.input_tokens	Prompt tokens (replaces gen_ai.usage.prompt_tokens)
gen_ai.usage.output_tokens	Completion tokens (replaces gen_ai.usage.completion_tokens)
gen_ai.response.finish_reasons	Array, e.g. ["stop"] or ["length"]
gen_ai.request.temperature, gen_ai.request.max_tokens	Request parameters
gen_ai.response.id	Provider-side completion id, for support tickets

The recommended span name is {operation} {model}, for example chat gpt-4o, with span kind CLIENT. Note that these conventions are still marked experimental, so pin your instrumentation version and expect names to keep shifting.

Wrapping a provider call in a span

Here is a Node example that instruments a single chat completion by hand. Hand-rolling it once is worth doing before you reach for an auto-instrumentation library, because it shows exactly what every wrapper is doing under the hood.

import { trace, metrics, SpanStatusCode, SpanKind } from "@opentelemetry/api";
import OpenAI from "openai";

const tracer = trace.getTracer("genai");
const meter = metrics.getMeter("genai");

// Histograms follow the GenAI metric conventions.
const tokenUsage = meter.createHistogram("gen_ai.client.token.usage", {
  unit: "{token}",
  description: "Tokens used per GenAI request",
});
const opDuration = meter.createHistogram("gen_ai.client.operation.duration", {
  unit: "s",
  description: "GenAI operation latency",
});

const openai = new OpenAI();

export async function chat(prompt: string) {
  const model = "gpt-4o";
  return tracer.startActiveSpan(
    `chat ${model}`,
    { kind: SpanKind.CLIENT },
    async (span) => {
      const start = performance.now();
      span.setAttributes({
        "gen_ai.provider.name": "openai",
        "gen_ai.operation.name": "chat",
        "gen_ai.request.model": model,
        "gen_ai.request.temperature": 0.2,
      });

      try {
        const res = await openai.chat.completions.create({
          model,
          temperature: 0.2,
          messages: [{ role: "user", content: prompt }],
        });

        const usage = res.usage;
        const attrs = { "gen_ai.request.model": model };

        span.setAttributes({
          "gen_ai.response.model": res.model,
          "gen_ai.response.id": res.id,
          "gen_ai.response.finish_reasons": [res.choices[0].finish_reason],
          "gen_ai.usage.input_tokens": usage?.prompt_tokens ?? 0,
          "gen_ai.usage.output_tokens": usage?.completion_tokens ?? 0,
        });

        // gen_ai.token.type separates input from output on the same histogram.
        tokenUsage.record(usage?.prompt_tokens ?? 0, {
          ...attrs,
          "gen_ai.token.type": "input",
        });
        tokenUsage.record(usage?.completion_tokens ?? 0, {
          ...attrs,
          "gen_ai.token.type": "output",
        });

        span.setStatus({ code: SpanStatusCode.OK });
        return res.choices[0].message.content;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        opDuration.record((performance.now() - start) / 1000, {
          "gen_ai.request.model": model,
        });
        span.end();
      }
    },
  );
}

Note what this buys you: the finally block records latency and ends the span even when the provider throws, so a timeout is a real ERROR span with a duration, not a silent gap.

Metrics: cost, tokens, and latency as first-class signals

Spans are for debugging one request. Histograms are for watching the whole fleet. The two metrics above give you three dashboards for free.

• Token usage (gen_ai.client.token.usage, unit {token}): split by the gen_ai.token.type dimension into input and output. Multiply by your per-token price to get a live cost estimate per model. This is where you catch the “quietly doubled the system prompt” regression.
• Operation duration (gen_ai.client.operation.duration, unit s): a latency histogram you can slice by gen_ai.request.model. This is where a provider degradation shows up as a p99 spike hours before anyone files a ticket.

Cost is not an OTel metric, it is a derived one. Keep the price table in your own config and compute input_tokens * input_price + output_tokens * output_price in your dashboarding layer, because prices change more often than your code deploys.

Correlating retrieval and generation in one trace

Most real features are not a single call. A RAG endpoint embeds the query, searches a vector store, then generates. If those live in separate spans under one parent, you can finally answer “was it slow retrieval or slow generation?” from a single trace.

sequenceDiagram
    participant U as User
    participant API as /answer handler
    participant VDB as Vector store
    participant LLM as Model provider

    U->>API: question
    Note over API: span: answer (SERVER, root)
    API->>VDB: embed + search
    Note over API,VDB: span: embeddings (CLIENT)
    VDB-->>API: top-k chunks
    API->>LLM: chat + context
    Note over API,LLM: span: chat gpt-4o (CLIENT)
    LLM-->>API: completion + usage
    API-->>U: answer

Because all three spans share one trace id, a slow answer decomposes instantly: 40ms embedding, 2.9s generation tells you where to spend effort. The same context propagation that makes distributed backends debuggable is doing the work here; nothing about it is LLM-specific.

Do not hand-roll everything: tools built on OTel

Once the pattern is clear, adopt a library so you are not decorating every call site by hand.

• OpenLLMetry (by Traceloop) is OTel-native auto-instrumentation for OpenAI, Anthropic, LangChain and more. It emits standard gen_ai.* spans to any OTLP endpoint.
• OpenInference (by Arize) is a parallel span convention feeding Phoenix, with OTel exporters.
• Langfuse ingests OTLP directly, so the spans from the code above can land in Langfuse without a separate SDK.

All of them speak OTLP, which means your LLM traces flow into the same collector as your HTTP and database spans. That matters when your agent starts calling tools, as in building an MCP server in TypeScript, and each tool call becomes its own child span in the same trace.

Takeaway

An uninstrumented LLM feature is a metered black box you are billed for and cannot debug. Wrap every model call in a span, set the gen_ai.* attributes, and record the two histograms, then let it flow through the OTel pipeline you already own. The moment you can slice latency by model, sum tokens by prompt version, and open one trace that spans retrieval and generation, “the summarize button sometimes spins forever” stops being a mystery and becomes a query. Instrument first, optimize second. You cannot fix what you cannot see.

References