Instrumenting a Wayfinding API with OpenTelemetry

The mechanics behind Wayfinding API Observability: where to put spans, which attributes are safe to aggregate on, and the sampling policy that keeps the interesting requests without storing all of them.

The Span Tree

The span tree for one routing request A sequence across five participants representing spans. The HTTP layer starts the root span and propagates the trace id. The route span opens a child span for snapping the origin, which returns an edge id or a refusal. It opens a second child for the graph solve, carrying the fallback tier as an attribute, which returns a node sequence. A third child generates instructions and returns the steps along with the instruction gap measurement. One request, four spans, three attributable stages HTTP route snap.origin graph.solve instructions span starts, trace_id propagated child span edge id, or refused child span, tier attribute node sequence child span steps + instruction_gap Four spans per request: enough to attribute latency, few enough to keep the trace readable.

Four spans is the right granularity. Fewer and a slow request is unattributable; more and every trace is a wall of spans nobody reads.

A routing request has three stages that can be slow for unrelated reasons, so each gets its own span:

  • snap.origin is a spatial index query plus a projection. When it is slow, the index is wrong-sized or the tolerance is admitting too many candidates.
  • graph.solve is the shortest-path search. When it is slow, the graph grew, or the request fell through to a fallback tier that re-solves on a modified copy.
  • instructions turns a node sequence into steps. When it is slow, the route is unusually long — which is itself worth knowing.

Attributes go on the root span so a trace can be filtered without expanding it, and the tier goes on the solve span because that is where it is decided. The instruction gap — the largest distance between consecutive turn instructions — is recorded here rather than computed later, because the generator already knows it and nothing downstream does.

Minimal Working Example 

import logging
import time

from opentelemetry import metrics, trace
from opentelemetry.trace import Status, StatusCode

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

tracer = trace.get_tracer("wayfinding.api")
meter = metrics.get_meter("wayfinding.api")

h_route = meter.create_histogram("route.duration", unit="s")
c_tier = meter.create_counter("route.tier")
h_gap = meter.create_histogram("route.instruction_gap", unit="m")
c_unsnapped = meter.create_counter("position.unsnapped")


def handle_route(req) -> dict:
    """Serve one routing request with the spans and metrics the service is judged on."""
    dims = {"profile": req.profile, "building": req.building, "map_version": req.map_version}
    started = time.perf_counter()

    with tracer.start_as_current_span("route") as root:
        for key, value in dims.items():
            root.set_attribute(f"route.{key}" if key == "profile" else key, value)

        with tracer.start_as_current_span("snap.origin") as snap_span:
            origin = snap(req.origin, req.building, level=req.level)
            snap_span.set_attribute("snap.refused", origin is None)
            if origin is None:
                c_unsnapped.add(1, dims)
                root.set_status(Status(StatusCode.ERROR, "origin could not be snapped"))
                raise OriginNotOnGraph(req.origin)

        with tracer.start_as_current_span("graph.solve") as solve_span:
            result = router.solve(origin, req.destination, profile=req.profile)
            solve_span.set_attribute("route.tier", result.tier)
            solve_span.set_attribute("route.length_m", round(result.length, 1))

        with tracer.start_as_current_span("instructions"):
            steps = make_instructions(result.path)
            gap = max((s.distance_m for s in steps), default=0.0)
            h_gap.record(gap, dims)

        c_tier.add(1, {**dims, "tier": str(result.tier)})
        h_route.record(time.perf_counter() - started, dims)
        return {"route": result.payload, "steps": steps,
                "profile_applied": req.profile, "tier": result.tier,
                "map_version": req.map_version}

Note that map_version appears in the response as well as in the telemetry. That is what turns a user report into a query: someone saying “it sent me the wrong way this morning” can be resolved against a specific build if the client recorded which one it was talking to.

Attributes and Cardinality

Which attributes belong on spans, on metrics, or on spans only A table of seven attributes. Map version, route profile, route tier and building id all belong on both spans and metrics, because their cardinality is low — around fifty map versions a year, four profiles, five tiers and a few hundred buildings — and each answers a question that averages hide. Trace id, origin coordinates and feature id belong on spans only: their cardinality is unbounded and using any of them as a metric dimension causes a cardinality explosion. Low cardinality on metrics, anything on spans Attribute Where Cardinality Why map.version span + metric ● low (~50/yr) attributes a regression to a release route.profile span + metric ● low (4) step-free regressions hide in averages route.tier span + metric ● low (5) the earliest degradation signal building.id span + metric ● low (~300) estates fail per building trace_id span only △ unbounded △ never a metric dimension origin coords span only △ unbounded △ never a metric dimension feature_id span only △ unbounded △ never a metric dimension The line between the top four and the bottom three is the difference between a bill and a surprise.

Cardinality is the only rule that matters here. Spans are sampled so unbounded attributes are free; metrics are aggregated across every series, so one unbounded dimension multiplies the whole set.

Metrics are aggregated across every combination of their dimensions, so each dimension multiplies the series count. Four low-cardinality dimensions — profile (4), tier (5), building (~300) and map version (~50 a year) — could in principle produce 300,000 series, but in practice a building only sees a handful of versions and profiles, so the realised count is a few thousand. That is comfortable.

Adding one unbounded dimension is not. trace_id on a metric produces one series per request; feature_id produces one per room per request. Both will be silently accepted by the SDK and will arrive as an invoice.

The safe rule is that anything identifying a request or an object belongs on a span, and anything identifying a category can be a metric dimension. Spans are sampled, so an unbounded attribute on a span costs storage proportional to the sample rate rather than to the cardinality.

Sampling Policy

What each sampling rate costs and catches Two curves against the base trace sampling rate. Stored traces per week rise linearly from 1,500 at one percent to 148,000 at full sampling. The share of slow requests captured rises steeply at first — 62 percent at one percent sampling, 84 at three percent, 96 at ten — then flattens, because tail-based rules already capture the interesting requests regardless of the base rate. 3% of everything plus 100% of what matters 0 50 100 150 25 50 75 100 base sampling rate (%) traces (k) / capture (%) traces stored per week (thousands) slow requests captured (%) 3% base + 100% on tier > 1 captures 84% of slow requests

A low base rate plus targeted 100% rules is the whole trick. Sampling every request that hits a fallback tier captures the interesting minority; the base rate only has to cover the ordinary case.

The policy that works is a low base rate plus targeted rules:

def should_sample(result, base_rate: float = 0.03) -> bool:
    # Sample ordinary requests sparsely and interesting ones always.
    if result.tier > 1:                 # any degradation is worth a full trace
        return True
    if result.seconds > 0.25:           # anything slow, regardless of tier
        return True
    if result.snap_refused:             # every refusal, they are rare and diagnostic
        return True
    return _deterministic_hash(result.trace_id) < base_rate

A 3% base rate stores about 4,500 traces a week at 150k requests, and the targeted rules add roughly the same again — because tier-2-and-above requests are about 3% of traffic by design. The result captures the overwhelming majority of the requests anyone would want to look at, for a fraction of the storage of full sampling.

The one thing to avoid is head-based sampling with no tail rules, which decides before the request runs and therefore cannot preferentially keep the slow ones. If the backend supports tail sampling, use it; if not, the deterministic-hash approach above with explicit rules gets most of the benefit.

Common Errors & Fixes

Traces stop at the API boundary. Context is not propagated into the worker or the graph service. Use the SDK’s context propagation rather than passing a trace id manually, and instrument outbound calls so the child spans attach to the right parent.

Metric cardinality explodes after a release. Someone added a useful-looking dimension. The fix is a lint in review: metric dimensions come from a fixed allow-list, and anything not on it is a span attribute.

Latency looks fine and users complain. The percentile is being computed across all profiles. Step-free routes are both slower and rarer, so they are invisible in an aggregate p95. Break every latency metric down by profile — that is why profile is a dimension.

The tier counter is always 1. The fallback tier is being recorded from the request rather than from the result, so it always reads as the tier that was attempted. Record it from the result, on the solve span, where the router actually decided it.

Integration Point

This instrumentation produces the raw material for everything in Wayfinding API Observability: the tier counter feeds the degradation share, the unsnapped counter feeds the map-quality band, and the instruction gap feeds route quality. The route quality guide turns those into the derived measures teams actually alert on.

Downstream, the same signals are what the rollback trigger watches — which is why map_version has to be a dimension rather than a log line. A trigger that cannot compare the current version against the previous one has nothing to roll back to.

Frequently Asked Questions

How many spans should one request produce?

Three to five for a routing request, and the test is whether each one answers a distinct question. Snapping, solving and instruction generation fail for unrelated reasons and have unrelated latency profiles, so separating them means a slow request is attributable at a glance. Going finer — a span per graph relaxation, say — produces traces nobody reads and measurable overhead, since span creation is not free. Going coarser leaves you knowing only that the request was slow.

Should the client be instrumented too?

Yes for the trace id, cautiously for everything else. Having the client generate and send a trace id, and having the API echo the map version back, is what connects a user report to a specific request — that alone is worth doing and carries no privacy weight. Full client-side tracing of positioning and rendering is a larger undertaking with real privacy considerations, since position traces are location history, and is better kept as an opt-in diagnostic mode than as always-on telemetry.

What retention makes sense for wayfinding traces?

Two weeks for traces, a year for metrics, and longer for probe results. Traces answer questions about specific recent requests and are almost never consulted after a fortnight, while being the expensive part of the bill. Metrics are cheap and are used for release-over-release and seasonal comparisons, which need a year. The synthetic probe results deserve their own long retention despite being tiny, because a year of them shows which buildings regress repeatedly — which is an estates conversation rather than an engineering one.

This page is a companion to Wayfinding API Observability, part of the Production-Ready Indoor Map Deployment section.