Tenant-Aware Observability with OpenTelemetry: Baggage, Spans, and Metric Cardinality

When a customer reports that their dashboard was slow at 9:30, your system-wide latency graph rarely helps. The average looks fine because that customer is one of thousands. To answer, you need to find that tenant's requests, follow them across services, and compare them with everyone else's.

That means attaching the tenant to telemetry, but not everywhere in the same way. This guide explains where a tenant ID belongs (traces and logs) and where it usually does not (metric labels), how to carry it across services with OpenTelemetry, and what to sample. The lab answers a question that many tutorials get wrong: if you put the tenant in W3C baggage, does it appear on your spans? It does not, unless you add something.

Where the tenant ID belongs

The metrics row is the one to be careful with. Prometheus's instrumentation guidance suggests keeping the cardinality of a metric below 10 where possible, and investigating alternatives for anything that could exceed 100. A tenant label on a request histogram multiplies its series by your number of tenants, times endpoints, times status codes, times buckets.

Use metrics to see that something is wrong, labeled by coarse dimensions such as plan tier, region, and endpoint group. Use traces and logs, filtered by tenant, to see who it affects. If a few very large tenants need their own dashboards, give them an explicit, bounded label value (cohort="tenant-acme") and put everyone else in cohort="other".

Carrying the tenant across services

You want the tenant ID, determined once at the edge from authenticated context, to be available in every downstream service without passing it through every function. OpenTelemetry offers two mechanisms, and they do different things:

• Trace context (traceparent) links spans into one trace.
• Baggage (baggage header) carries arbitrary key-value pairs alongside the trace context to downstream services.

OpenTelemetry's baggage documentation says plainly that baggage is not the same as attributes: it is a separate store, and values must be explicitly read from baggage and added to spans, metrics, or logs if you want them there.

Lab: does baggage show up on spans?

The program starts a "gateway" span, puts tenant.id=tnt_42 into baggage, and calls a downstream "billing" service over HTTP through a transport that injects trace context and baggage. Every finished span is recorded in memory. The run is repeated with a span processor that copies one allow-listed baggage key onto each span when it starts.

In an empty directory, run go mod init lab, save the program as main.go, then go mod tidy && go run ..

// Lab: does W3C baggage show up on spans by itself?
//
// An ingress handler puts tenant.id into baggage, then calls a downstream
// service over HTTP using a client transport that injects trace context and
// baggage. We record every finished span in memory and print which spans
// carry a tenant.id attribute, first with a default tracer provider and then
// with a span processor that copies one allow-listed baggage key at span start.
package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/baggage"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/sdk/trace/tracetest"
)

// tenantFromBaggage copies only tenant.id from baggage onto new spans.
type tenantFromBaggage struct{}

func (tenantFromBaggage) OnStart(ctx context.Context, s sdktrace.ReadWriteSpan) {
	if v := baggage.FromContext(ctx).Member("tenant.id").Value(); v != "" {
		s.SetAttributes(attribute.String("tenant.id", v))
	}
}
func (tenantFromBaggage) OnEnd(sdktrace.ReadOnlySpan)      {}
func (tenantFromBaggage) Shutdown(context.Context) error   { return nil }
func (tenantFromBaggage) ForceFlush(context.Context) error { return nil }

// propagating injects trace context and baggage into outbound requests.
type propagating struct{ base http.RoundTripper }

func (p propagating) RoundTrip(r *http.Request) (*http.Response, error) {
	r = r.Clone(r.Context())
	otel.GetTextMapPropagator().Inject(r.Context(), propagation.HeaderCarrier(r.Header))
	return p.base.RoundTrip(r)
}

func run(copyBaggage bool) {
	rec := tracetest.NewSpanRecorder()
	opts := []sdktrace.TracerProviderOption{sdktrace.WithSpanProcessor(rec)}
	if copyBaggage {
		opts = append([]sdktrace.TracerProviderOption{sdktrace.WithSpanProcessor(tenantFromBaggage{})}, opts...)
	}
	tp := sdktrace.NewTracerProvider(opts...)
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
	tr := tp.Tracer("lab")

	var seenHeader string
	downstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		seenHeader = r.Header.Get("baggage")
		ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
		_, span := tr.Start(ctx, "billing: load invoice")
		span.End()
	}))
	defer downstream.Close()

	// ingress
	m, _ := baggage.NewMember("tenant.id", "tnt_42")
	b, _ := baggage.New(m)
	ctx := baggage.ContextWithBaggage(context.Background(), b)
	ctx, root := tr.Start(ctx, "gateway: GET /invoices/9")
	_, child := tr.Start(ctx, "gateway: authorize")
	child.End()
	client := &http.Client{Transport: propagating{http.DefaultTransport}}
	req, _ := http.NewRequestWithContext(ctx, http.MethodGet, downstream.URL, nil)
	resp, err := client.Do(req)
	if err == nil {
		io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
	}
	root.End()

	fmt.Printf("downstream received header  baggage: %q\n", seenHeader)
	for _, s := range rec.Ended() {
		val := "(none)"
		for _, a := range s.Attributes() {
			if a.Key == "tenant.id" {
				val = a.Value.AsString()
			}
		}
		fmt.Printf("  span %-28q tenant.id attribute: %s\n", s.Name(), val)
	}
	tp.Shutdown(context.Background())
}

func main() {
	fmt.Println("1) default tracer provider")
	run(false)
	fmt.Println("\n2) with a span processor that copies tenant.id from baggage")
	run(true)
}

Output recorded on November 3, 2026 (Go 1.26.5, windows/amd64, OpenTelemetry Go v1.47.0):

1) default tracer provider
downstream received header  baggage: "tenant.id=tnt_42"
  span "gateway: authorize"         tenant.id attribute: (none)
  span "billing: load invoice"      tenant.id attribute: (none)
  span "gateway: GET /invoices/9"   tenant.id attribute: (none)

2) with a span processor that copies tenant.id from baggage
downstream received header  baggage: "tenant.id=tnt_42"
  span "gateway: authorize"         tenant.id attribute: tnt_42
  span "billing: load invoice"      tenant.id attribute: tnt_42
  span "gateway: GET /invoices/9"   tenant.id attribute: tnt_42

What the output shows

• Baggage propagated. The downstream service received baggage: "tenant.id=tnt_42" in both runs.
• No span carried the tenant by default. Not the gateway spans, not the downstream span. A trace search for tenant.id = tnt_42 would find nothing.
• With the span processor, every span carried it, including the downstream span, because the processor reads baggage from the context each span starts in.

The OpenTelemetry Go contrib repository includes a ready-made processor for this, baggagecopy, which takes a filter function. Use a filter. Copying every baggage key onto every span lets any upstream caller decide what gets written into your telemetry.

The previous version of this guide claimed that downstream spans "automatically inherit" the tenant from baggage, and its round-tripper passed the propagator itself as the carrier: Inject(req.Context(), otel.GetTextMapPropagator()). That line does not compile; Go reports that propagation.TextMapPropagator does not implement propagation.TextMapCarrier. The lab's propagating transport shows the working form, which injects into propagation.HeaderCarrier(r.Header).

Getting the tenant into every log line

Logs need the same tenant ID, plus the trace ID so a log line can be joined to its trace. Passing both to every log call is tedious and easy to forget. With Go's log/slog, a small handler wrapper can read them from the context instead. The program below sets the tenant once after authentication and logs three lines: one before authentication, one after, and one through a call that does not pass the context. Fixed trace IDs make the output reproducible. In an empty directory run go mod init lab, save it as main.go, then go mod tidy && go run ..

// Lab: put the tenant and trace IDs on every log line without passing them
// to each log call. A slog.Handler wrapper reads them from the context.
package main

import (
	"context"
	"log/slog"
	"os"

	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

type tenantKey struct{}

// WithTenant is called once, after authentication.
func WithTenant(ctx context.Context, id string) context.Context {
	return context.WithValue(ctx, tenantKey{}, id)
}

// contextHandler adds tenant.id, trace_id and span_id from the context.
type contextHandler struct{ slog.Handler }

func (h contextHandler) Handle(ctx context.Context, r slog.Record) error {
	if id, ok := ctx.Value(tenantKey{}).(string); ok {
		r.AddAttrs(slog.String("tenant.id", id))
	}
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(slog.String("trace_id", sc.TraceID().String()), slog.String("span_id", sc.SpanID().String()))
	}
	return h.Handler.Handle(ctx, r)
}

func (h contextHandler) WithAttrs(a []slog.Attr) slog.Handler {
	return contextHandler{h.Handler.WithAttrs(a)}
}
func (h contextHandler) WithGroup(n string) slog.Handler {
	return contextHandler{h.Handler.WithGroup(n)}
}

func main() {
	opts := &slog.HandlerOptions{ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
		if a.Key == slog.TimeKey {
			return slog.Attr{} // omit time so the output is reproducible
		}
		return a
	}}
	log := slog.New(contextHandler{slog.NewJSONHandler(os.Stdout, opts)})

	tp := sdktrace.NewTracerProvider(sdktrace.WithIDGenerator(fixedIDs{}))
	ctx, span := tp.Tracer("lab").Start(context.Background(), "GET /invoices/9")
	defer span.End()

	log.InfoContext(ctx, "before authentication")
	ctx = WithTenant(ctx, "tnt_42")
	log.InfoContext(ctx, "invoice loaded", "invoice", "in_9")
	log.Info("logged without the context") // the handler cannot see tenant or trace here
}

// fixedIDs makes trace and span IDs reproducible for the recorded output.
type fixedIDs struct{}

func (fixedIDs) NewIDs(context.Context) (trace.TraceID, trace.SpanID) {
	return trace.TraceID{0x4b, 0xf9, 0x2f, 0x35, 0x77, 0xb3, 0x4d, 0xa6, 0xa3, 0xce, 0x92, 0x9d, 0x0e, 0x0e, 0x47, 0x36},
		trace.SpanID{0x00, 0xf0, 0x67, 0xaa, 0x0b, 0xa9, 0x02, 0xb7}
}
func (fixedIDs) NewSpanID(context.Context, trace.TraceID) trace.SpanID {
	return trace.SpanID{0x00, 0xf0, 0x67, 0xaa, 0x0b, 0xa9, 0x02, 0xb8}
}

Output recorded on November 3, 2026 (Go 1.26.5, OpenTelemetry Go v1.47.0):

{"level":"INFO","msg":"before authentication","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}
{"level":"INFO","msg":"invoice loaded","invoice":"in_9","tenant.id":"tnt_42","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}
{"level":"INFO","msg":"logged without the context"}

The first line has trace IDs but no tenant, because authentication had not run yet. The second has both. The third has neither: log.Info without a context gives the handler nothing to read. Use the ...Context variants (InfoContext, ErrorContext) everywhere inside request handling, and a linter rule that flags the context-free calls there, or tenant fields will be missing exactly in the code paths that log errors in a hurry.

Baggage is input, so treat it that way

Baggage travels in a plain HTTP header. Anything that can send a request to your service can set it, and your service forwards it to everything downstream, including third-party APIs, unless you stop it. OpenTelemetry's documentation lists this among its security considerations.

• Set the tenant in baggage at the edge, after authentication, and overwrite any value the client sent.
• Strip incoming baggage at your public edge if you do not use it from outside.
• Do not make authorization decisions from baggage. Downstream services should get the tenant from authenticated context, such as a verified token, and use baggage only to label telemetry.
• Do not put personal data in baggage. An internal tenant ID is fine; an email address is not.

Background jobs and message queues

HTTP propagation stops at a queue unless you put the context into the message. Inject trace context and baggage into message headers when publishing, and extract them when consuming, using the same propagator. The job queue guide covers carrying the tenant itself in the job payload, which is the authoritative copy for the worker; the propagated context is for telemetry only.

Sampling

Recording every trace is often too expensive. Head sampling decides at the start of a trace, before you know whether it will be slow or fail. Tail sampling, in the OpenTelemetry Collector, waits until a trace completes and then applies policies. The tail sampling processor can keep traces with errors, traces above a latency threshold, traces matching attribute values, and a probabilistic share of the rest.

A starting configuration for a multi-tenant service might keep all error traces, all traces slower than your latency objective, all traces for tenants on plans with support commitments, and a small percentage of everything else. Policies are evaluated together, so a trace kept by any "keep" policy is kept; read the processor's documentation on how decisions combine, especially if you use and or composite policies. Tail sampling holds traces in memory until it decides, so size the collector for your trace volume and decision wait, and monitor the traces it drops.

Where to start

Search your traces for a tenant attribute. If you are relying on baggage and the search finds nothing, add a filtered baggage-copying span processor. Then check your metrics backend for series whose labels include a tenant or user ID, and replace them with tiers or bounded cohorts. Finally, make sure your public edge overwrites any baggage a client sends. The rate-limiting guide uses the same tenant-in-logs approach to find which tenants are being throttled.