Automating Rollback Triggers for Indoor Map Deploys

This page covers the mechanism that reverts a bad indoor-map release automatically — watching live telemetry and, when it breaches a threshold, repointing the active map at the last-known-good artifact — and it sits inside Rollback Triggers & Versioning, the safety net downstream of every promotion.

What a Telemetry-Driven Rollback Trigger Means

A rollback trigger is an automated decision: given live signals from the field, either keep serving the current map release or revert to the previous known-good one. It exists because some faults only appear in production. A dataset can pass every pre-deploy gate — valid geometry, one connected component, correct door adjacency — and still degrade navigation because the positioning system behaves differently against the new geometry, or a corridor was mapped open that is physically closed. Those show up as behaviour, not as a schema error, so the only place to catch them is the running fleet.

The trigger consumes three kinds of signal that clients already emit: the route-failure rate (queries that returned no path), the positioning degradation-stage distribution (how often clients fall back down the ladder described in SDK Integration Patterns), and the general client error rate. When those breach a baseline by more than a set margin, over a set window, the trigger flips the active pointer — a map_version / topology_hash reference — back to the last artifact that was healthy. Crucially, it reverts to a gated artifact: the target has already passed the topology checks in CI, so the revert lands on something known-navigable, never on an unvalidated guess.

The three states of an automated rollback trigger Three states. The service is healthy while serving version 42. When a monitored metric crosses its threshold it moves to watching, where the breach is observed but not yet acted on. If the breach is sustained for five minutes the trigger fires and the service rolls back to version 41 with caches purged. If the metric recovers inside the window the service returns to healthy without any rollback, which is what stops a single traffic spike from reverting a good map. Breach, confirm, act — the middle step prevents most rollbacks metric crosses threshold sustained 5 min recovers inside the window Healthy serving v42 Watching breach observed, not yet confirmed Rolled back serving v41, caches purged

The watching state is the whole design. Without it, every transient spike reverts the map — and a service that rolls back on noise is one nobody will let deploy automatically.

Minimal Working Example

A single decision function. It takes the current live signals, a health baseline, and the last-known-good reference, checks each signal against its threshold over the window, and — if any breach and the release is out of its cooldown — returns a revert to the known-good artifact. Every path is logged so the decision is auditable.

import logging
from dataclasses import dataclass

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


@dataclass
class Signals:
    route_failure_rate: float      # fraction of route queries returning no path
    degraded_stage_rate: float     # fraction of clients past the "latest synced graph" stage
    client_error_rate: float       # fraction of sessions reporting a client error


@dataclass
class Thresholds:
    max_route_failure: float = 0.03
    max_degraded_stage: float = 0.10
    max_client_error: float = 0.05


def evaluate_rollback(
    current_hash: str,
    live: Signals,
    baseline: Signals,
    last_known_good: str,
    thr: Thresholds,
    in_cooldown: bool,
) -> str | None:
    """Return the topology_hash to revert to, or None to hold the current release."""
    breaches = []
    if live.route_failure_rate - baseline.route_failure_rate > thr.max_route_failure:
        breaches.append(f"route_failure +{live.route_failure_rate - baseline.route_failure_rate:.3f}")
    if live.degraded_stage_rate - baseline.degraded_stage_rate > thr.max_degraded_stage:
        breaches.append(f"degraded_stage +{live.degraded_stage_rate - baseline.degraded_stage_rate:.3f}")
    if live.client_error_rate - baseline.client_error_rate > thr.max_client_error:
        breaches.append(f"client_error +{live.client_error_rate - baseline.client_error_rate:.3f}")

    if not breaches:
        logger.info("release %s healthy; holding", current_hash[:12])
        return None
    if in_cooldown:
        logger.warning("release %s breached (%s) but in cooldown; not flapping", current_hash[:12], ", ".join(breaches))
        return None
    if last_known_good == current_hash:
        logger.error("release %s breached but last-known-good == current; manual intervention", current_hash[:12])
        return None

    logger.error("ROLLBACK %s -> %s : %s", current_hash[:12], last_known_good[:12], ", ".join(breaches))
    return last_known_good

The return value is the topology_hash the deployment controller repoints the active pointer to — the same content-addressed identity used across the delivery path — or None to hold. The controller performs the pointer swap; this function only makes the call, which keeps the decision testable in isolation.

Rollback Trigger Reference

Parameter Meaning Typical value Notes
route_failure_rate Fraction of route queries returning no path baseline + 3% The most direct signal a floor became unnavigable
degraded_stage_rate Fraction of clients past the latest-synced stage baseline + 10% Rising fallback down the SDK degradation ladder
client_error_rate Fraction of sessions reporting an error baseline + 5% Catches parse/render faults the other two miss
window Evaluation window before deciding 5–15 min Long enough to be statistically meaningful, short enough to limit blast radius
cooldown Minimum time between rollback actions ≥ 30 min Prevents flapping between two releases
last_known_good The revert target previous gated topology_hash Must be a gated artifact, never an unvalidated build
baseline Pre-release healthy signal levels rolling prior-week median Compare against health, not against zero
The six signals worth rolling a map deployment back on A table of six rollback signals with thresholds and confirmation windows. A route solve failure rate above two percent for five minutes is the user-visible symptom. A p95 route latency above 400 milliseconds for five minutes indicates a graph that grew pathologically. A tile 404 rate above half a percent for three minutes indicates a partial publish. Any schema validation error at all, confirmed over one minute, is never acceptable in production. An unsnapped fix rate above eight percent over ten minutes means the graph geometry moved under live users. A client crash rate above 0.3 percent for five minutes means an envelope some client cannot parse. Six signals, each with a window sized to its ambiguity Signal Threshold Window Why this metric route_solve_failure_rate > 2% 5 min the user-visible symptom p95 route latency > 400 ms 5 min a graph that grew pathologically tile 404 rate > 0.5% 3 min a partial publish schema_validation_errors > 0 1 min △ never acceptable in production unsnapped_fix_rate > 8% 10 min graph geometry moved under the users client crash rate > 0.3% 5 min an envelope a client cannot parse Windows are shorter where the signal is unambiguous and longer where traffic mix matters.

Window length encodes ambiguity. A schema error means one thing and gets one minute; an unsnapped-fix spike could be a busy lobby, so it gets ten.

Common Errors & Fixes

The trigger flaps between two releases. Without a cooldown, the controller reverts to the old version, sees the new baseline is still elevated, reverts again, and oscillates — each swap forcing clients to re-sync. Add a cooldown and hysteresis so a revert cannot immediately be undone and a marginal breach does not toggle repeatedly:

A real rollback: failure rate, confirmation window, and recovery A single curve of route solve failure rate over eighteen minutes. The rate sits at about 0.3 percent while version 43 rolls out, then jumps to 3.9 percent six minutes after it reaches full traffic, crossing the two percent threshold. It stays between 3.9 and 4.4 percent for the next five minutes, which satisfies the confirmation window, and the rollback fires at minute twelve. By minute fourteen version 41 is restored and the rate is back to 0.5 percent, returning to baseline shortly after. Threshold at minute 6, rollback at minute 12, recovered by 14 0 1 2 3 4 0 5 10 15 minutes since v43 reached 100% of traffic route solve failure rate (%) threshold crossed sustained 5 min -> rollback fires v41 restored

Eight minutes of degraded service, no human involved. The same incident handled by paging an on-call engineer typically runs 30-45 minutes — which is the entire argument for automating the trigger.

if in_cooldown:
    logger.warning("breach within cooldown window; suppressing to avoid flapping")
    return None  # let the window elapse before any further action

Rolling back to an equally-bad version. The trigger reverts to last_known_good, but that reference was set to a release that was itself already degraded, so navigation does not recover. Only advance the last-known-good pointer to a release after it has held healthy for a full window, never at promotion time:

def maybe_promote_known_good(current_hash: str, held_healthy_for_min: int, window_min: int) -> str | None:
    if held_healthy_for_min >= window_min:
        return current_hash        # this release earned known-good status
    return None                    # not yet proven; keep the previous known-good

No health baseline, so every deploy looks broken. Comparing live signals against zero flags normal background route-failure and fallback as a regression, and the trigger reverts good releases. Compare against a rolling baseline — the prior week’s median for that building — so the threshold measures change caused by the release, not the absolute rate, which is never zero for real indoor positioning.

Integration Point

This trigger closes the loop that the rest of the deployment lifecycle opens. Its input signals are exactly the telemetry the client emits while degrading — the degradation_stage counters and route-failure metrics defined in SDK Integration Patterns — so the same fail-closed ladder that protects an individual user also feeds the fleet-wide revert decision. Its revert target is an artifact that already cleared CI Gating for Map Updates, so the automatic rollback can only ever land on a dataset proven navigable at build time. Pre-deploy gating and post-deploy rollback are the two halves of the same guarantee: a user is never left on a map that cannot route them.

Frequently Asked Questions

How do I stop the rollback from flapping between two versions?

Enforce a cooldown and hysteresis. A cooldown is a minimum interval — 30 minutes or more — during which no further rollback action is allowed after one fires, so the system cannot revert and immediately un-revert. Hysteresis means the threshold to roll back is higher than the level you consider recovered, so a signal hovering near the boundary does not toggle the decision on every evaluation. Together they turn a noisy signal into at most one action per incident.

What should the trigger roll back to?

The last release that both preceded the current one and held healthy for a full window — a gated topology_hash that already passed the topology checks in CI. Never revert to an arbitrary previous build or to a version that was itself degraded when it was live. Advance the last-known-good pointer only after a release has proven healthy in production, not at promotion time, so the revert target is always a genuinely good map rather than just an older one.

Which signal is the strongest rollback trigger?

The route-failure rate is the most direct — a query that returns no path on a floor that previously routed is an unambiguous regression. But it lags, because it needs users to actually attempt affected routes. The positioning degradation-stage distribution moves earlier: a rising share of clients falling past the latest-synced graph means the release is failing before route requests pile up. Watch all three against a baseline; the degradation-stage distribution is usually your earliest warning.

This page is part of the Rollback Triggers & Versioning guide within the Production-Ready Indoor Map Deployment reference.