Code changes: likely culprits and review priority

Overview

AI coding tools mean far more pull requests merge to production every day. When something breaks (main goes red, errors spike, an incident is declared), "what changed recently?" is no longer a short list you can read in a minute. It is forty PRs across six services, many of them large, many reviewed quickly.

We answer two questions over that change stream:

1. Attribution, after a breakage: which merged PRs most likely contributed to this, and why?
2. Priority, before merge: which open PRs deserve a careful human review, based on what has broken before?

We built it for teams that merge a lot (often with coding agents) and already revert, write postmortems or close incident issues with "Caused by #123". Both questions use the same learned links, and that feedback trains them.

TOP 3 CAUSES OF error_spike FOR checkout AT now() EXPLAIN;

1  PR #4107  Add idempotency_key column             0.407
   merged 40m before · migration 0.50 · review:fast 0.33 · 140 lines
2  PR #4108  Update lockfile                         0.290
   merged 30m before · lockfile 0.10 · dep_bump 0.10 · 900 lines
3  PR #4103  Refactor payment retry client           0.004
   merged 3h before · kernel 0.01, too early for an error spike

How it maps

Features

Design notes

• Explanations point at changes, never at people. Features describe the PR (what it touched, how big, how it was reviewed), never who wrote it. HeapDB ranks changes to review or revert; it is not a scorecard for engineers or for tools.
• Credit lands on the real cause. AI-generated PRs are often large and approved quickly, so ai_authored rides along with riskier features. Proportional credit and joint causes keep its link low in the simulation; feature interactions such as "a migration with a rushed review" are on the roadmap.
• The revert confirms. A ranked list with evidence tells you where to look first; the revert or the postmortem confirms it and becomes feedback.

Setup

Everything below runs on the open-source heapdb-core library: pip install heapdb (a Rust engine with a Python API; prebuilt wheels for Python 3.10+, no runtime dependencies).

Run the example

examples/code_changes.py simulates a week of about 150 PRs across services with a hidden ground truth: merges, deploys, reverts as feedback, and a pre-merge review queue.

pip install heapdb
curl -O https://raw.githubusercontent.com/heapdb/heapdb-core/main/examples/code_changes.py
python code_changes.py      # needs only heapdb; no clone or Rust toolchain

Embed the library

pip install heapdb

import math
from heapdb import HeapDB, ExponentialDecay, LogNormalKernel
 
M, H = 60, 3600
db = HeapDB(bucket_seconds=15 * M)
 
# One effect per way a breakage shows up, each with its own lag kernel.
db.register_effect("failing_main_check", kernel=ExponentialDecay(half_life=10 * M), cause_types=["deploy"])
db.register_effect("error_spike", kernel=LogNormalKernel(median=30 * M, sigma=0.8), cause_types=["deploy"],
                   priors={"migration": (3, 3), "review:fast": (2, 4), "ai_authored": (1, 6)},
                   default_prior=(1, 9))
db.register_effect("incident", kernel=LogNormalKernel(median=3 * H, sigma=0.7), cause_types=["deploy"])
 
# Dose = log1p(lines changed). docs_only can never be a culprit.
db.register_event_type("deploy", weight=lambda a: math.log1p(a["lines_changed"]) / math.log1p(100))
db.add_rule("error_spike", "docs-only change", lambda e: "docs_only" in e.features, 0.0)
 
t0 = 12 * H
db.insert("checkout", t0 + 0 * M, "deploy", {"path:payments/retry", "size:large", "ai_authored"},
          attrs={"pr": 4103, "lines_changed": 640})
pr4107 = db.insert("checkout", t0 + 140 * M, "deploy",
                   {"migration", "path:payments/retry", "review:fast", "ai_authored"},
                   attrs={"pr": 4107, "lines_changed": 140})
pr4108 = db.insert("checkout", t0 + 150 * M, "deploy", {"lockfile", "dep_bump", "size:large"},
                   attrs={"pr": 4108, "lines_changed": 900})
db.insert("checkout", t0 + 160 * M, "deploy", {"docs_only"}, attrs={"pr": 4109, "lines_changed": 120})
 
spike = t0 + 180 * M
for c in db.top_causes("checkout", "error_spike", at=spike, k=3, explain=True):
    print(c.rank, c.event.attrs["pr"], round(c.score, 3), c.explanation)
 
# PR #4107 was reverted; the postmortem cleared #4108.
rep = db.feedback("checkout", "error_spike", at_time=spike,
                  confirmed_event_id=pr4107.id, rejected_ids=[pr4108.id])
print(rep.credit)   # migration 0.67, review:fast 0.45, ai_authored 0.19
 
# Score an open PR before it merges (no lag kernel).
risk = db.feature_risk("error_spike", {"migration", "path:payments/retry", "ai_authored"},
                       entity="checkout", type="deploy", attrs={"lines_changed": 300})

With a connector (available on request)

The GitHub connector runs this loop on a real repository. It receives GitHub App webhooks (merged PRs, deployments, failing checks, incident issues) and alert webhooks, and turns them into the inserts and effects above. Reverts and incident issues closed with "Caused by #123" become feedback automatically, and the other suspects are rejected. When main goes red or errors spike, it posts the likely culprits as a check run or comment; on open PRs it posts the pre-merge risk score as a comment, updated on each push. It is available on request: see Connectors for what it needs and how to get access.

Explanation

Reading an EXPLAIN line

lag 0.67h -> kernel 0.511 x weight 1.07 x link 0.743 [noisy-OR of migration=0.50 (a=3, b=3), review:fast=0.33 (a=2, b=4), ...]

• lag → kernel: PR #4107 merged 40 minutes before the spike; the error_spike kernel (median 30 min) gives that 0.511. A PR merged three hours earlier gets 0.01.
• weight: the dose, from 140 lines changed.
• link: noisy-OR over the PR's features. Each feature's strength is a Beta mean a / (a + b); the counts say how sure it is.

Feedback that comes for free

• A revert of PR #4107 within a few hours of an effect confirms #4107 for that effect.
• An incident issue labelled incident whose body says Caused by #4107 confirms it too.
• PRs cleared in the postmortem (or simply the other top candidates once a culprit is confirmed) are rejected.

Confirmations use proportional credit: each feature of the reverted PR gains credit in proportion to how much it already explained the effect.

Priority: which open PR should a human review carefully?

The same links score a PR that has not merged yet. There is no lag, so no kernel:

RISK OF incident FOR checkout FEATURES migration, path:payments/retry, review:fast WEIGHT 1.2 EXPLAIN;

The GitHub connector (available on request) posts this as a PR comment, updated in place, so a reviewer facing twenty AI-generated PRs knows which three to read line by line. RISK OF, db.feature_risk(...) and db.rank_candidates(...) are available from heapdb-core v0.3.

Benchmarks

Ranking quality on heapdb-core's examples/code_changes.py simulation (a week of about 150 PRs, hidden ground truth), seeds 0-19. Rank of the true culprit PR; mean ± sample std over seeds. Copied from the committed benchmark results. recency = most recent eligible deploy first; kernel_only = kernel × dose × rules, all links equal; priors_only = HeapDB with the example's priors and no feedback; heapdb_proportional / heapdb_uniform = HeapDB with revert feedback and EM vs uniform credit.

hit@1

hit@3

Before any feedback, the kernel, dose and noisy-OR over feature sets take whole-week hit@1 from 0.367 (recency) to 0.536 (priors only), and revert feedback raises it to 0.562. Methodology note: the simulator's lags match the configured kernels, which favours kernel-based methods over recency more than real-world lags would.

Latency: the Rust engine (0.5) scores 19.2 of the 459 events in the window at 1M events; its p50 is 0.107 ms, 16× faster than SQLite in memory (1.75 ms). Synthetic data, one small VM. Full numbers and methodology: Benchmarks.