Incidents: which change caused the spike?

Overview

When an alert fires, on-call needs a short list. Of everything that changed on this service (deploys, config edits, flag flips), what most likely explains the spike? Code changes starts from PRs and GitHub. This example starts from the alert and ranks mixed change types, most of which never went through a PR.

We built it for SRE and on-call teams whose monitoring (Sentry, PagerDuty, Datadog, Grafana) can call a webhook, and who write postmortems.

Top causes of 'latency_spike' for checkout:
  1. config [db_pool_size,timeout]   score=0.2691
      lag 0.42h -> kernel 0.420 x weight 1 x link 0.640 [noisy-OR of db_pool_size=0.60 (a=3, b=2), timeout=0.10 (a=1, b=9)]
  2. flag [promo_banner]             score=0.0379
      lag 0.13h -> kernel 0.758 x weight 1 x link 0.100 x rules flags are gated x0.5
  3. config [db_pool_size]           score=0.0357
      lag 0.25h -> kernel 0.595 x weight 1 x link 0.600 x rules dry-run config x0.1
  [segments: 3 scanned / 4 pruned / 7 in window; events scored: 5]

That's the output of heapdb-core's examples/incident_rca.py. It covers a week of 234 changes, and HeapDB answers by scoring only 5 of them.

How it maps

Design notes

• Send every change source. HeapDB ranks the changes it receives, so feed it deploys, config edits and flag flips from every system that makes them.
• One effect per alert id. Repeats with the same id are deduplicated; give monitors that fire for the same incident a shared id.
• Feedback moves the ranking. A well-timed but innocent change drops as postmortems reject it.

Setup

Everything below runs on the open-source heapdb-core library: pip install heapdb (a Rust engine with a Python API; prebuilt wheels for Python 3.10+, no runtime dependencies).

Run the example

pip install heapdb
curl -O https://raw.githubusercontent.com/heapdb/heapdb-core/main/examples/incident_rca.py
python incident_rca.py      # needs only heapdb; no clone or Rust toolchain

Embed the library

pip install heapdb

import math
from heapdb import ExponentialDecay, HeapDB, format_result
 
M, H = 60.0, 3600.0
db = HeapDB(bucket_seconds=10 * M)
db.register_event_type("deploy", weight=lambda a: math.log1p(a.get("lines_changed", 100)) / math.log1p(100))
db.register_effect(
    "latency_spike",
    kernel=ExponentialDecay(half_life=20 * M),
    default_prior=(1, 9),                                   # most changes are innocent
    priors={"db_pool_size": (3, 2), "cache_ttl": (2, 3), "orm_upgrade": (2, 3), "log_level": (1, 20)},
    cause_types={"deploy", "config", "flag"},               # cron is never a cause
)
db.add_rule("latency_spike", "dry-run config", lambda e: bool(e.attrs.get("dry_run")), 0.1)
db.add_rule("latency_spike", "flags are gated", None, 0.5, types={"flag"})
 
# Insert deploys, config changes and flag flips as they happen.
t0 = 10 * H
db.insert("checkout", t0, "deploy", {"orm_upgrade"}, attrs={"lines_changed": 300})
db.insert("checkout", t0 + 2 * H, "config", {"db_pool_size", "timeout"})
db.insert("checkout", t0 + 2 * H + 10 * M, "config", {"db_pool_size"}, attrs={"dry_run": True})
db.insert("checkout", t0 + 2 * H + 17 * M, "flag", {"promo_banner"})
db.insert("checkout", t0 + 2 * H + 20 * M, "cron", {"log_rotate"})
 
spike_time = t0 + 2 * H + 25 * M
print(format_result(db.top_causes("checkout", "latency_spike", at=spike_time, k=3, explain=True)))

Call db.insert(...) from wherever your deploys, config changes and flag flips happen, and top_causes when an alert fires. When the postmortem names the culprit, db.feedback(...) confirms it and rejects the other suspects.

With a connector (available on request)

The generic alert webhook lets a Sentry alert rule, PagerDuty webhook or Datadog monitor post each alert as an effect, answered with the top likely causes. Together with the GitHub connector, merged PRs and deploys arrive as changes automatically, and reverts and "Caused by #123" on incident issues become feedback. Config and flag events from other systems are on the connectors roadmap; until then, insert them with the library as above. It is available on request: see Connectors for details and how to get access.

Explanation

• The kernel narrows, the links decide. A config change 25 minutes before the spike gets kernel 0.420; a flag flip 8 minutes before gets 0.758. The flag still ranks below the config change because its link is weak (0.10) and the gated-rollout rule halves it.
• Rules carry what you already know. A dry-run config change never touched production, so it is multiplied by 0.1 and the rule is named in the explanation. cause_types removes cron jobs entirely and tightens pruning bounds.
• Group queries rank the same spike across services: top_causes_group(["checkout", "payments", "search"], ...). Scores are comparable across services only once calibrated.
• Postmortem feedback confirms the culprit with proportional credit (db_pool_size 0.94, timeout 0.16 in the example) and also updates a global layer, so a brand-new service starts from the lesson (its db_pool_size link starts at 0.66).
• Live ticker. With an exponential kernel the score has the forward-decay form, so a ForwardDecayHeap keyed by dose × link × rules keeps the hottest recent change on top at every instant without rescoring.

Benchmarks