Automating Rollback Triggers for Indoor Map Deploys

This page covers the mechanism that reverts a bad indoor-map release automatically — watching live telemetry and, when it breaches a threshold, repointing the active map at the last-known-good artifact — and it sits inside Rollback Triggers & Versioning, the safety net downstream of every promotion.

What a Telemetry-Driven Rollback Trigger Means

A rollback trigger is an automated decision: given live signals from the field, either keep serving the current map release or revert to the previous known-good one. It exists because some faults only appear in production. A dataset can pass every pre-deploy gate — valid geometry, one connected component, correct door adjacency — and still degrade navigation because the positioning system behaves differently against the new geometry, or a corridor was mapped open that is physically closed. Those show up as behaviour, not as a schema error, so the only place to catch them is the running fleet.

The trigger consumes three kinds of signal that clients already emit: the route-failure rate (queries that returned no path), the positioning degradation-stage distribution (how often clients fall back down the ladder described in SDK Integration Patterns), and the general client error rate. When those breach a baseline by more than a set margin, over a set window, the trigger flips the active pointer — a map_version / topology_hash reference — back to the last artifact that was healthy. Crucially, it reverts to a gated artifact: the target has already passed the topology checks in CI, so the revert lands on something known-navigable, never on an unvalidated guess.

Minimal Working Example

A single decision function. It takes the current live signals, a health baseline, and the last-known-good reference, checks each signal against its threshold over the window, and — if any breach and the release is out of its cooldown — returns a revert to the known-good artifact. Every path is logged so the decision is auditable.

import logging
from dataclasses import dataclass

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


@dataclass
class Signals:
    route_failure_rate: float      # fraction of route queries returning no path
    degraded_stage_rate: float     # fraction of clients past the "latest synced graph" stage
    client_error_rate: float       # fraction of sessions reporting a client error


@dataclass
class Thresholds:
    max_route_failure: float = 0.03
    max_degraded_stage: float = 0.10
    max_client_error: float = 0.05


def evaluate_rollback(
    current_hash: str,
    live: Signals,
    baseline: Signals,
    last_known_good: str,
    thr: Thresholds,
    in_cooldown: bool,
) -> str | None:
    """Return the topology_hash to revert to, or None to hold the current release."""
    breaches = []
    if live.route_failure_rate - baseline.route_failure_rate > thr.max_route_failure:
        breaches.append(f"route_failure +{live.route_failure_rate - baseline.route_failure_rate:.3f}")
    if live.degraded_stage_rate - baseline.degraded_stage_rate > thr.max_degraded_stage:
        breaches.append(f"degraded_stage +{live.degraded_stage_rate - baseline.degraded_stage_rate:.3f}")
    if live.client_error_rate - baseline.client_error_rate > thr.max_client_error:
        breaches.append(f"client_error +{live.client_error_rate - baseline.client_error_rate:.3f}")

    if not breaches:
        logger.info("release %s healthy; holding", current_hash[:12])
        return None
    if in_cooldown:
        logger.warning("release %s breached (%s) but in cooldown; not flapping", current_hash[:12], ", ".join(breaches))
        return None
    if last_known_good == current_hash:
        logger.error("release %s breached but last-known-good == current; manual intervention", current_hash[:12])
        return None

    logger.error("ROLLBACK %s -> %s : %s", current_hash[:12], last_known_good[:12], ", ".join(breaches))
    return last_known_good

The return value is the topology_hash the deployment controller repoints the active pointer to — the same content-addressed identity used across the delivery path — or None to hold. The controller performs the pointer swap; this function only makes the call, which keeps the decision testable in isolation.

Rollback Trigger Reference

Parameter Meaning Typical value Notes
route_failure_rate Fraction of route queries returning no path baseline + 3% The most direct signal a floor became unnavigable
degraded_stage_rate Fraction of clients past the latest-synced stage baseline + 10% Rising fallback down the SDK degradation ladder
client_error_rate Fraction of sessions reporting an error baseline + 5% Catches parse/render faults the other two miss
window Evaluation window before deciding 5–15 min Long enough to be statistically meaningful, short enough to limit blast radius
cooldown Minimum time between rollback actions ≥ 30 min Prevents flapping between two releases
last_known_good The revert target previous gated topology_hash Must be a gated artifact, never an unvalidated build
baseline Pre-release healthy signal levels rolling prior-week median Compare against health, not against zero

Common Errors & Fixes

The trigger flaps between two releases. Without a cooldown, the controller reverts to the old version, sees the new baseline is still elevated, reverts again, and oscillates — each swap forcing clients to re-sync. Add a cooldown and hysteresis so a revert cannot immediately be undone and a marginal breach does not toggle repeatedly:

if in_cooldown:
    logger.warning("breach within cooldown window; suppressing to avoid flapping")
    return None  # let the window elapse before any further action

Rolling back to an equally-bad version. The trigger reverts to last_known_good, but that reference was set to a release that was itself already degraded, so navigation does not recover. Only advance the last-known-good pointer to a release after it has held healthy for a full window, never at promotion time:

def maybe_promote_known_good(current_hash: str, held_healthy_for_min: int, window_min: int) -> str | None:
    if held_healthy_for_min >= window_min:
        return current_hash        # this release earned known-good status
    return None                    # not yet proven; keep the previous known-good

No health baseline, so every deploy looks broken. Comparing live signals against zero flags normal background route-failure and fallback as a regression, and the trigger reverts good releases. Compare against a rolling baseline — the prior week’s median for that building — so the threshold measures change caused by the release, not the absolute rate, which is never zero for real indoor positioning.

Integration Point

This trigger closes the loop that the rest of the deployment lifecycle opens. Its input signals are exactly the telemetry the client emits while degrading — the degradation_stage counters and route-failure metrics defined in SDK Integration Patterns — so the same fail-closed ladder that protects an individual user also feeds the fleet-wide revert decision. Its revert target is an artifact that already cleared CI Gating for Map Updates, so the automatic rollback can only ever land on a dataset proven navigable at build time. Pre-deploy gating and post-deploy rollback are the two halves of the same guarantee: a user is never left on a map that cannot route them.

Frequently Asked Questions

How do I stop the rollback from flapping between two versions?

Enforce a cooldown and hysteresis. A cooldown is a minimum interval — 30 minutes or more — during which no further rollback action is allowed after one fires, so the system cannot revert and immediately un-revert. Hysteresis means the threshold to roll back is higher than the level you consider recovered, so a signal hovering near the boundary does not toggle the decision on every evaluation. Together they turn a noisy signal into at most one action per incident.

What should the trigger roll back to?

The last release that both preceded the current one and held healthy for a full window — a gated topology_hash that already passed the topology checks in CI. Never revert to an arbitrary previous build or to a version that was itself degraded when it was live. Advance the last-known-good pointer only after a release has proven healthy in production, not at promotion time, so the revert target is always a genuinely good map rather than just an older one.

Which signal is the strongest rollback trigger?

The route-failure rate is the most direct — a query that returns no path on a floor that previously routed is an unambiguous regression. But it lags, because it needs users to actually attempt affected routes. The positioning degradation-stage distribution moves earlier: a rising share of clients falling past the latest-synced graph means the release is failing before route requests pile up. Watch all three against a baseline; the degradation-stage distribution is usually your earliest warning.

This page is part of the Rollback Triggers & Versioning guide within the Production-Ready Indoor Map Deployment reference.