Skip to content
Running AI in production· Guide 9 of 14Intermediate8 min readUpdated

Tracing vs logging for LLM and agent apps ​

A log line records one event: "request completed", "model call failed", "tool timed out". A trace records one request as a tree of timed steps, called spans, with parents and children. In a plain web app the difference is a matter of convenience. In an LLM or agent app it decides whether you can answer the basic questions. One user request turns into several model calls, the tool calls the model asked for, the servers behind those tools, and sometimes a queue job or a run resumed after a person approves a step. Logs show each event on its own. A trace shows which call caused which, and where the time and tokens went. You need both, joined by a trace id on every log line. Keep prompts, answers and tool arguments out of both by default.

Side by side ​

LogsTraces
UnitOne event, as a line or a recordOne request, as a tree of spans
OpenTelemetry's definition"A recording of an event.""The path of a request through your application."
AnswersWhat happened, and with what error?What caused what, in which order, and how long did each step take?
StructureFlat; related lines share an idParent and child spans; links between related traces
Crosses servicesOnly if every service logs the same idYes, through the W3C traceparent and tracestate values (HTTP headers, or message metadata)
Typical volume controlLog levelsSampling: at the start (head) or after the trace is complete (tail)
Good forErrors with details, start-up settings, background jobs, anything outside a requestLatency breakdown, fan-out, tool chains, work that hops between services
Weak atReconstructing a whole agent runFree-text detail; events that belong to no request

Metrics are the third signal: counts and histograms, such as tokens per model or tool-call latency, that stay cheap however much traffic you have. They answer "how often" and "how much", not "what happened in this run".

One agent run as log lines and as a traceOn the left, four flat log lines, each with the same trace id. On the right, the same run as a tree: an agent span containing two model call spans and a tool call span, with an MCP server call under the tool call.Logs: one line per event10:02:01 run startedtrace_id=4bf9…10:02:03 tool lookup_order oktrace_id=4bf9…10:02:09 model call slowtrace_id=4bf9…10:02:09 run completedtrace_id=4bf9…Trace: one tree per requestinvoke_agent support-bot (8.1 s)chat (1.4 s)execute_tool lookup_order (0.6 s)tools/call → MCP serverchat (6.0 s, the slow step)
The same run twice. The log lines share a trace id, so you can find them all, but only the trace shows that the second model call took most of the eight seconds.

What logs do well ​

Logs are cheap to write, easy to search and need no special backend. They are the right place for:

  • Errors with their details. A stack trace, a provider's error message or a configuration problem fits a log line, not a span attribute.
  • Things that belong to no request. Start-up settings, a scheduler firing, a retention purge, a lost connection to the database.
  • A quick look. grep on a request id still beats any query language when a service is down.

Write them as structured records, such as JSON with fixed field names, not free text. OpenTelemetry defines a structured log as "a log with a defined, consistent schema or typed fields". Then every line can carry the ids that tie it to the rest of the system.

What traces do well ​

An agent's run is a chain of decisions, and its cost and latency come from that chain. A trace answers questions a pile of log lines cannot:

  • Where the time went. Was the run slow because of the model, a tool, the MCP server behind the tool, or a queue wait?
  • Where the tokens went. With the token counts on each model call span, you can see which turn of the loop used the most context.
  • Which call caused which. The model asked for a tool, the tool called a server, and the server's error came back as the next prompt. Parent and child spans show this order.
  • Work that leaves the process. With W3C Trace Context, which "defines standard HTTP headers and a value format to propagate context information", the trace continues into an MCP server or a queue worker that supports it: each hop carries a traceparent value with the same trace id, in an HTTP header, in the MCP request's params._meta, or in the queued job's payload. Work that resumes much later, such as a run continued after an approval, joins with a span link rather than a parent.

The OpenTelemetry GenAI semantic conventionsGenAI semantic conventions: OpenTelemetry's shared names for the spans, attributes and metrics of model calls, agents and tool calls, such as `chat`, `invoke_agent` and `gen_ai.usage.input_tokens`. give these spans shared names, such as chat {model}, execute_tool {tool} and invoke_agent {agent}, so any tracing backend can draw an agent timeline. Mapping your traces to them covers the names in detail.

What to capture, and what to keep out ​

The same rule applies to both signals: record ids, names, numbers and outcomes; keep content out unless you have a reason and a plan.

Capture on every model call: provider, requested and returned model, input and output tokens, finish reason, duration, retry count, error type. On every tool call: tool name, call id, the server it went to, outcome (ok, error, refused, waiting for approval), duration. On the run: the ids that link it to your own records, so someone can go from a trace to the run and back.

Keep out by default: prompts, model answers, tool arguments and tool results. They hold personal dataPII: Personally identifiable information: data that identifies a person alone or combined with other data, such as a name, an email address or an ID number., customer documents and sometimes secrets, and tracing and log backends are usually read by more people, and kept longer, than your database. The GenAI conventions agree: their content attributes are Opt-In, and the 1.37.0 release notes say they "are not recorded by default when content capturing is disabled". When you do need content to debug an agent, turn it on deliberately, mask it before export, restrict who can read it and keep it briefly. Keeping personal data out of prompts, logs and traces covers masking.

Keep ids off metrics. A run id or user id as a metric attribute creates a new series for every run. Put those on spans and log lines.

Sampling changes the picture ​

Logs are usually all kept, filtered by level. Traces are often sampled. OpenTelemetry describes head sampling as a decision made "as early as possible", without seeing the whole trace, and tail sampling as a decision made "by considering all or most of the spans within the trace". Head sampling at 10% throws away 90% of failed runs too. If you sample, keep every error trace with tail sampling in a collector, and make sure the logs of a dropped trace still carry its trace id, so the lines can be grouped even when the spans are gone.

When to use which ​

  • A run failed and you need the reason: start from the logs or the run's error, then open its trace to see which step failed and what led up to it.
  • Runs are slow or expensive: use traces for one run and metrics for the trend.
  • You need proof of who changed what: neither. That is an audit logAudit log: A record of who did what and when, such as changes to settings, rules and secrets, kept so that actions can be checked later., with its own retention and protection; see audit trails for AI agents.
  • You need the full inputs and outputs of a run: your app's own run history, under its access control and retention, not a tracing backend.

How they combine ​

The join is the trace id. OpenTelemetry's logging guidance says that with its SDK or auto-instrumentation in place, OpenTelemetry can "automatically correlate your existing logs with any active trace and span", which means each log line written inside a span carries trace_id and span_id. In practice:

  1. Every log line written inside a request or run carries trace_id and span_id.
  2. Your run records store the trace id, and the screen that shows a run links to its trace.
  3. Spans carry your run id, so a trace leads back to the run.
  4. Metrics stay aggregate and point to traces through exemplars or dashboards, not ids.

With that in place you can start anywhere (an alert, a support ticket, a slow-request graph) and reach the other two in one click.

In L2M ​

L2M, a self-hosted workflow builder and agent runtime, keeps the three signals separate and joins them by id (Observability):

  • Logs are JSON lines on stdout, with reqId on every line of a request (also returned as X-Request-ID). LOG_LEVEL is warn by default; info adds request and run start and end lines.
  • Metrics are on a Prometheus /metrics page, on by default, with run counts, durations and SLO numbers. They can also be pushed over OTLP.
  • Traces are optional and off by default. With an OTLP endpoint set, L2M emits spans for each HTTP request, run, node, agent loop, model call, tool call and MCP request, following the GenAI and MCP conventions pinned at v1.41.0 (OpenTelemetry). Spans carry ids and timings, not content. Content capture is a separate opt-in that masks before export (content capture).
  • The join. While tracing is on, log lines written inside a request or run carry trace_id and span_id. Each run stores its trace id, shown as Copy trace id on the run page, and L2M_OTEL_TRACE_URL_TEMPLATE adds an Open trace link. Spans carry l2m.execution.id to lead back to the run (trace id on runs).
  • Records that are neither. Run history keeps each run's inputs, outputs and node trace inside L2M, under EXECUTION_HISTORY_RETENTION_DAYS. MCP tool-call telemetry keeps one row per tool call with a safe argument summary. The audit log records who changed the platform.

Try it. The quick start with a local collector runs the zero-key MCP Agent Quickstart template and writes its spans to a file. Start the API with LOG_LEVEL=info as well, and the run's log lines carry the same trace id as its spans.

Further reading ​

Last updated:

This website (not the L2M product) uses Google Analytics, only after you accept cookies. Cookie settings