How do I set up distributed tracing (OpenTelemetry) across microservices?
Question
A request travels API Gateway → Auth → Order → Stock. We're struggling to find the source of an 800ms delay in one of the services in between. With OpenTelemetry we plan to carry a `trace_id` across all services (Laravel and Go) and visualize it in a central APM (Jaeger/Grafana Tempo). How do I design context propagation over HTTP headers and queue messages?
Answer
Short answer
Don’t invent a custom scheme; standardize on W3C Trace Context (traceparent/tracestate) so PHP and Go interoperate without friction.
Context
Your real problem is “where’s the 800ms” — and to see it the trace has to travel across services and queues without breaking. I covered the Go-side HTTP mechanics, i.e. how context travels with the request, in writing HTTP services in Go.
Why
-
A standard header is the common ground between two languages. Because W3C Trace Context is a standard, Laravel and Go read the same
traceparent. Invent your owntrace_idheader and you owe a translation layer for every new service. -
The chain breaks most often at the queue hop. This is the piece teams forget: get the HTTP side right but fail to carry context when publishing a message, and async hops show up as orphan traces in the APM. I discussed the durability side of that same queue separately in the question about buffering the log pipeline.
-
Head-only sampling drops exactly the trace you’re after. If you want to see the 800ms outlier, the start of the request is the wrong place to decide.
What to do
-
Make W3C Trace Context the standard. Start the root span at the gateway; every service extracts context from the incoming HTTP header and injects it into outgoing calls.
-
Carry the same
traceparentthrough queues. When you publish a message, addtraceparentas a message header/attribute and re-extract it in the worker. -
Put the OTel SDK in each service, ship via OTLP. Service → OTLP → a Collector → Jaeger/Tempo. A Collector in between decouples export from the app and makes swapping backends easy.
-
Choose sampling smartly. Use
parent-basedso all spans of a trace share the same fate; add tail sampling on top so slow traces are definitely kept. -
Don’t hand-roll trace_id. Don’t write your own header read/write code; the SDK’s context propagation already carries it. Do it by hand and you’ll forget it somewhere and the trace snaps there.
Bottom line: I’d set up W3C Trace Context + the OTel SDK + a Collector + Tempo/Jaeger, leave propagation to the SDK, and never skip putting traceparent into the message header on the queue hop. For the case where the message format is frozen and cannot carry a traceparent, see Distributed Tracing Across a Polyglot Queue, where I tie the trace to the message’s trace_id instead. Only an unbroken trace shows you the 800ms.
Compiled from questions I have run into in the field or been asked.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.