Wayfinding API Observability
A wayfinding service can return 200 for every request, in twelve milliseconds, while routing people to a floor that closed last month. Standard application monitoring cannot see that, because nothing about it is an error. This topic, part of Production-Ready Indoor Map Deployment, covers the instrumentation that can: what to measure, what a breach of each signal means, and the synthetic probes that find a stranded wing before a visitor does.
The Problem: Healthy and Wrong Are Indistinguishable
Generic observability answers one question well — is the service responding, and how fast. For a wayfinding service that is the least interesting of three questions, because the two above it fail silently.
Map quality failures produce correct-looking responses over wrong data. A wing whose lift was never modelled is unroutable, and the API reports that as a normal “no route” outcome; a level whose geometry moved under live users produces a rising unsnapped-fix rate and no errors at all.
Route quality failures produce valid routes that people do not follow. A step-free route that is 3.5× the default route is technically correct and practically a refusal; a route with no turn instructions for 80 m is correct and unusable. Neither raises anything.
Service health is the layer every APM tool already covers, and it is genuinely necessary — a wayfinding API that is down is down. It is simply not sufficient, and a team that instruments only this layer finds out about the other two from support tickets.
Prerequisites & Dependencies
| Dependency | Version | Used for |
|---|---|---|
opentelemetry-sdk |
≥ 1.24 | spans, metrics, trace propagation |
opentelemetry-instrumentation-fastapi |
≥ 0.45 | automatic HTTP spans |
prometheus-client |
≥ 0.20 | metric export where OTLP is not available |
Three pieces of context have to be available at request time for any of this to be useful:
- The map version, from the content-addressed artifact the service is serving. Without it a metric regression cannot be attributed to a release.
- The profile applied, from accessible routing profiles. Route-quality metrics mean different things per profile and averaging across them hides the step-free problems entirely.
- The fallback tier, from fallback routing architectures. A route served by tier 3 is a successful response and a signal that something upstream is degraded.
The Signals Worth Alerting On
Two of these deserve elaboration because they are the ones teams most often lack.
unroutable_pairs comes from a synthetic probe rather than from traffic. A background job
routes between a fixed sample of origin-destination pairs — one per floor pair, plus every entrance
to every major destination — on every map publish and every hour thereafter. It is the only signal
that finds a stranded wing before someone tries to walk to it, and because the pairs are fixed,
a failure names the exact pair rather than a rate.
detour_ratio is the length of a step-free route divided by the default route for the same
pair. It is the accessibility metric that matters, and it is invisible in aggregate latency or
success metrics: a wheelchair user who is routed 3.5× further has been served successfully by every
conventional measure. Tracking it at p90 per building surfaces the buildings where lift coverage is
genuinely thin, which is an estates problem rather than a software one — but nobody can act on it
until it is measured.
Step-by-Step Implementation
Step 1 — span the parts that can be slow for different reasons.
import logging
from opentelemetry import metrics, trace
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
tracer = trace.get_tracer("wayfinding.api")
meter = metrics.get_meter("wayfinding.api")
route_seconds = meter.create_histogram("route.duration", unit="s")
tier_counter = meter.create_counter("route.tier")
detour_ratio = meter.create_histogram("route.detour_ratio")
def solve_route(req) -> dict:
"""One routing request, instrumented so each failure mode is separable."""
with tracer.start_as_current_span("route") as span:
span.set_attribute("map.version", req.map_version)
span.set_attribute("route.profile", req.profile)
span.set_attribute("building.id", req.building)
with tracer.start_as_current_span("snap.origin"):
origin = snap(req.origin, req.building)
span.set_attribute("snap.refused", origin is None)
with tracer.start_as_current_span("graph.solve"):
result = router.solve(origin, req.destination, profile=req.profile)
span.set_attribute("route.tier", result.tier)
tier_counter.add(1, {"tier": result.tier, "profile": req.profile})
route_seconds.record(result.seconds, {"profile": req.profile})
if req.profile != "default" and result.default_length:
ratio = result.length / result.default_length
detour_ratio.record(ratio, {"profile": req.profile, "building": req.building})
span.set_attribute("route.detour_ratio", round(ratio, 2))
return result.payloadStep 2 — run the synthetic reachability probe.
import itertools
def reachability_probe(graph, sample_pairs: list[tuple[str, str]],
profiles: dict[str, dict]) -> dict[str, list[tuple[str, str]]]:
"""Route a fixed sample of pairs per profile; report the ones that fail by name."""
if not sample_pairs:
raise ValueError("probe needs a fixed pair sample; a random one cannot be compared")
failures: dict[str, list[tuple[str, str]]] = {}
for name, weights in profiles.items():
broken = []
for a, b in sample_pairs:
try:
router.solve(a, b, profile=name, graph=graph)
except NoRouteFound:
broken.append((a, b))
if broken:
failures[name] = broken
logger.error("probe: %s cannot route %d/%d pair(s), e.g. %s -> %s",
name, len(broken), len(sample_pairs), *broken[0])
return failuresStep 3 — attribute every metric to a map version. The single most useful dashboard in a
wayfinding stack is any of these signals broken down by map.version, because it turns
“routing got worse this week” into “routing got worse in build 9f3c, which shipped on Tuesday”.
Validation & Failure Modes
| Skipped | What you lose | How it surfaces instead |
|---|---|---|
map.version on spans |
Attribution to a release | A week of bisecting by hand |
| Profile dimension | Step-free regressions | An accessibility complaint |
| Tier counter | Early warning of degradation | A latency spike, much later |
| Reachability probe | Stranded wings | A visitor at a locked corridor |
| Unsnapped-fix rate | Geometry drift under users | Rising reroutes, unexplained |
| Detour ratio | Thin lift coverage | Nothing — it never surfaces |
The last row is worth stating plainly. Detour ratio has no organic failure signal at all: users who are routed a long way around either walk it or give up, and neither produces an error. A service that does not measure it will never learn it has a problem, which is a different situation from the other rows, where the information arrives eventually and expensively.
A healthy weekly summary:
{
"map_version": "sha256:9f3c1a",
"routes": 148203,
"tier_share": {"1": 0.971, "2": 0.021, "3": 0.006, "4": 0.0015, "5": 0.0005},
"unsnapped_fix_rate": 0.031,
"reroutes_per_session": 0.42,
"detour_ratio_p90": {"step_free": 1.9, "low_vision": 1.2},
"probe": {"pairs": 412, "unroutable": {"default": 0, "step_free": 0}}
}Operational Considerations
Sample traces, count everything. Full tracing on a busy wayfinding API is expensive and unnecessary — a 1-5% trace sample answers “why was this request slow”, while the metrics, which are cheap, answer “how often does this happen”. The exception is traces for requests that hit a fallback tier beyond the first: those are rare and individually interesting, so they are worth sampling at 100%.
Cardinality is the trap. building.id is fine as a metric dimension — an estate has hundreds
of buildings, not millions. feature_id, trace_id and origin coordinates are not, and putting
any of them on a metric will produce a cardinality explosion that a metrics backend charges for.
They belong on spans, which are sampled, not on metrics, which are aggregated.
Alert on the probe, page on the service. A stranded wing needs a ticket for whoever maintains the map, not a 3 a.m. page for the on-call engineer — it has probably been broken since the last publish and will not get worse overnight. Latency and error-rate breaches are the opposite. Wiring both into the same alerting policy trains people to ignore the map alerts.
Keep one dashboard per audience. The three layers have three audiences: map quality belongs to whoever maintains the building data, route quality to whoever owns the product experience, and service health to whoever carries the pager. A single dashboard mixing all three is read carefully once and skimmed thereafter, because two thirds of it is somebody else’s problem. Splitting them also makes the ownership question explicit, which is what turns a rising detour ratio from an interesting chart into a work item on an estates backlog.
Feed the rollback trigger. The signals here are the same ones the rollback trigger watches, and connecting them closes the loop: a map publish that pushes the unroutable-pair count above zero or the tier-2 share past its threshold reverts itself rather than waiting for a human to notice.
Performance & Scale Notes
Telemetry has to cost less than the failures it catches, and for a wayfinding service the numbers work out comfortably — provided cardinality is controlled.
| Signal | Volume at 150k routes/week | Cost driver |
|---|---|---|
| Metrics (6 signals × 4 dimensions) | ~2,900 series | dimension cardinality |
| Traces at 3% sampling | ~4,500 traces/week | span count per request |
| Traces for tier > 1 at 100% | ~4,400 traces/week | rare by construction |
| Reachability probe | 412 solves/hour | graph solve time |
| Weekly aggregate report | 1 document | negligible |
The metric series count is the number to watch. Six signals across four dimensions — profile,
building, tier and map version — is a few thousand series, which every backend handles without
comment. Adding feature_id or trace_id as a dimension turns that into millions and is the
single most common way a well-intentioned observability project becomes an unexpected invoice.
The reachability probe is the only signal with real compute cost, and it is bounded by the pair sample rather than by traffic: 412 pairs across every profile is roughly 1,600 solves, about 12 seconds of CPU on a campus graph. Running it hourly and on every publish costs a few minutes of CPU a day, which is negligible against the cost of a stranded wing going unnoticed for a week.
Two practical economies are worth taking. Sample traces at a low rate for ordinary requests but at 100% for requests served beyond the first fallback tier — those are the interesting ones and, by design, there are few of them. And compute the detour ratio only for non-default profiles, since it is defined as a ratio against the default route and computing it for the default is both meaningless and doubles the solve count for every request.
Frequently Asked Questions
What is the single most valuable metric to add first?
The synthetic reachability probe, because it is the only one that finds the worst failure before a user does. Everything else measures traffic, and traffic only tells you about the parts of the building people are already trying to reach — a wing that has been unroutable since Tuesday produces no signal at all until someone attempts to walk to it. A fixed sample of origin-destination pairs routed on every publish costs seconds of compute and turns that silent failure into a named pair in a report. Add it before any dashboard.
How do I measure route quality without user feedback?
Through proxies that correlate well and cost nothing: the reroute rate, the detour ratio, and the distance between turn instructions. A session with three reroutes in two minutes usually means the position and the map disagree; a detour ratio above about 2.5 means the accessible route is one people are likely to abandon; an instruction gap of 80 m means the route is being described too sparsely to follow. None of these is a substitute for asking users, but all three move in the right direction when the experience is bad, and all three can be computed from data the service already has.
Should positioning quality be part of the same instrumentation?
The parts of it the API can see, yes — and it is one of the more useful cross-cutting signals. The unsnapped-fix rate is measured server-side and rises for two quite different reasons: the positioning got worse, or the graph moved under users after a publish. Breaking it down by map version separates them immediately, which is a diagnosis neither team could reach alone. Device-side positioning accuracy is a different pipeline with its own privacy considerations, and is better kept separate from route serving.
How long should wayfinding telemetry be retained?
Long enough to compare a release against the one before it, which in practice means aggregated metrics for a year and sampled traces for a fortnight. The comparisons that matter are seasonal — a building behaves differently in term time — and are made against aggregates, which are cheap. Traces are expensive and are only ever used to answer a question about a specific recent request, so retaining them beyond a couple of weeks buys very little. The probe results deserve their own longer retention, since a year of them shows which buildings repeatedly regress.
Related
- Instrumenting a Wayfinding API with OpenTelemetry — spans, attributes and the sampling policy in detail.
- Measuring Indoor Route Quality in Production — detour ratio, reroute rate and instruction gap as computable proxies.
- Alerting on Indoor Positioning Degradation — separating a positioning regression from a map regression.
- Rollback Triggers & Versioning — the consumer of these signals when a publish goes wrong.
- CI Gating for Map Updates — the same reachability question, asked before publication instead of after.
This page is part of the Production-Ready Indoor Map Deployment section.