APM and Distributed Tracing: OpenTelemetry in Practice

APM and Distributed Tracing: OpenTelemetry in Practice

Once a website evolves from a monolith to "frontend + gateway + microservices + database + third-party APIs", a single request crosses dozens of components. Service-level metrics can only tell you "this service is slow", not "which hop is slow". Distributed tracing reconstructs the full path of a request with a trace that spans all services, and OpenTelemetry (OTel) has become the de-facto standard here — a vendor-neutral, pluggable telemetry framework.

Core concepts: traces, spans, and context

Per the official OpenTelemetry docs, a trace is made of spans, each representing a unit of work and containing a name, parent span ID, start/end timestamps, span context, attributes, events, links, and status. All spans in the same trace share one trace_id, and the parent_id field expresses the hierarchy — that is exactly how traces are reconstructed.

Three details worth understanding:

• Span Context: the immutable object on every span holding the trace ID, span ID, trace flags, and trace state. It is the part serialized and propagated alongside distributed context.
• Span attributes: follow semantic convention naming so metadata is standardized across systems. Prefer adding attributes at span creation so they are available to SDK sampling.
• Span events vs. attributes: if a timestamp is meaningful, use a span event (e.g., the moment "the page becomes interactive"); otherwise use attributes.

Creating a span manually with the SDK is straightforward — in Python:

from opentelemetry import trace

tracer = trace.get_tracer("shop.checkout")
with tracer.start_as_current_span("charge_payment") as span:
    span.set_attribute("order_id", "20260808-001")
    span.set_attribute("payment.provider", "stripe")
    span.add_event("retry_after_timeout", {"attempt": 2})
    result = charge()
    span.set_status(trace.StatusCode.OK if result else trace.StatusCode.ERROR)

A few points: attributes like order_id should be set when the span is created so samplers can decide by order dimension; add_event is for moments with a timestamp, such as a retry; the root span is usually created automatically by the framework or an entry middleware, so business code only needs inner spans at critical call sites rather than manual instrumentation everywhere.

Context propagation and span kinds

Context propagation is what makes distributed tracing work: without it, spans generated by different services can't be assembled into one trace. In practice, span context travels from upstream to downstream via HTTP headers such as traceparent. OTel also defines span kinds — Client (outgoing synchronous remote call), Server (incoming remote call), Internal (in-process), and Producer/Consumer (async queue production and consumption) — which help backends assemble traces correctly.

Sampling: balancing cost and completeness

Full sampling is unaffordable at high traffic, so sampling is a key design decision. Two common approaches:

• Head sampling: decides at the request entry whether to sample. Simple and efficient, but it can't tell whether this particular request will turn out to be anomalous.
• Tail sampling: the Collector decides after spans finish, based on results (errors, slow requests). This guarantees all failing traces are kept, at the cost of buffering and a more complex deployment.

A pragmatic recommendation: keep 100% of errors and slow requests, sample normal requests at 5%-10%, and back this up with metrics — because what sampled traces can't show is exactly what the metrics layer should catch.

The Collector's tail sampling can be declared in a config file, with policies that keep "error" and "slow" traces:

processors:
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 5000}
  batch:

The trade-offs between the two sampling approaches:

Correlating the three pillars: metrics, logs, and traces

Tracing is not isolated — it only forms a complete troubleshooting loop when correlated with metrics and logs: metrics tell you "something is wrong" (error rate or latency rising), logs give the detail (the actual error stack), and traces locate the hop (which service, which call). OTel's context propagation ties these signals together naturally — put trace_id in logs and you can jump from one log line straight to its trace; put service dimensions on metrics and you can drill from a chart down to individual calls. The standard flow for debugging a slow request: confirm the anomaly window on a metrics dashboard, use traces to find the bottleneck service, then use that service's logs to pinpoint the root cause.

Locating one slow request

During a big promo, the P95 latency on a mall's campaign page jumped from 800ms to 3.2 seconds. After confirming the anomaly window on the metrics dashboard, an engineer opened a sampled slow trace and found the time was almost entirely inside an internal span named inventory.check_stock — not the payment gateway the team had suspected. Following that trace, they used the trace_id in the logs to find the exact SQL behind that call, spotting an inventory query that was missing an index. After adding the index, P95 dropped back to 900ms.

In this case the three signals did their jobs: metrics found the anomaly, traces located the hop, logs reconstructed the detail. Without traces, the team could easily have spent hours debugging the wrong component.

Reference: OpenTelemetry Traces concept docs https://opentelemetry.io/docs/concepts/signals/traces/; tail sampling configuration https://opentelemetry.io/docs/collector/configuration/#tail-sampling-processor

16IDC Take

For most independent sites, a full OTel rollout may be overkill, but two things are worth doing: instrument key services with tracing SDKs so "where is the slowness" becomes answerable, and use the OTel Collector as a single entry point to export metrics, logs, and traces together, avoiding vendor lock-in. To correlate logs and traces see log aggregation and query, for the frontend journey see RUM monitoring, and for a lighter error-tracking alternative see Sentry error monitoring. For the infrastructure layer pair it with Prometheus + Grafana. See more in the Monitoring & Alerting category.

Source: https://opentelemetry.io/docs/concepts/signals/traces/