Product analytics: what goes with outcomes?

Overview

Product teams want to know how what users do and see relates to outcomes: which feature usage goes with conversion, which errors or slow pages go with rage clicks, support tickets or churn. Nobody labels "this caused that", so unlike code changes there are no reverts to learn from. So HeapDB learns observationally. Each outcome gives soft credit to the events in its kernel window, and windows that pass without the outcome count as evidence against.

We built it for product and growth teams on PostHog who want a ranked, explained shortlist of hypotheses, with uncertainty, to test with experiments.

--- converted: 754 occurrences, baseline 0.012
+ event:onboarding_completed          0.155 [0.135, 0.176]  13.18     829  yes
+ page:/home                          0.045 [0.039, 0.051]   3.80    2836  yes
+ event:used_dashboard_export         0.017 [0.011, 0.025]   1.48     882  no
  ! possible confounder: event:used_dashboard_export (association only; confirm with an experiment)

Measured output of a PostHog-format replay (PostHog connector, synthetic data; columns: strength, 90% interval, lift, support, interval clears baseline). used_dashboard_export was planted as a confounder: power users use it and convert more, but it has no effect of its own.

How it maps

Priors are weak and sceptical (Beta(1, 9), mean 0.1): most events cause nothing. A background rate (leak) is larger for effects that often happen for reasons outside the stream, such as churn.

Design notes

• Pair it with experiments. Observational links mix cause with selection (power users do everything). HeapDB shows where to look; an A/B test confirms an effect, and is the right tool for small effects such as a few percent change in conversion.
• Confounders are kept in check. Proportional credit keeps a feature that only rides along with a real cause below it.
• Scale. The Rust engine (0.5) was measured up to 10M events (benchmarks), and heapdb-server runs it as a network service with a write-ahead log and snapshots.
• Privacy. Forgetting a person is supported (forget_entity); more controls are on the roadmap.

Setup

Everything below runs on the open-source heapdb-core library: pip install heapdb (a Rust engine with a Python API; prebuilt wheels for Python 3.10+, no runtime dependencies).

Run the example

examples/product_analytics.py simulates 1,200 users with feature flags, pages, onboarding, exceptions and a hidden power-user confounder, then prints a link report per outcome next to the planted truth.

pip install heapdb
curl -O https://raw.githubusercontent.com/heapdb/heapdb-core/main/examples/product_analytics.py
python product_analytics.py      # needs only heapdb; no clone or Rust toolchain

Embed the library

pip install heapdb

from heapdb import HeapDB, LogNormalKernel
 
M, H, D = 60.0, 3600.0, 86400.0
db = HeapDB(bucket_seconds=6 * H)
# One-shot outcome: after a person converts, later events get no negative evidence.
db.register_effect("converted", LogNormalKernel(median=D, sigma=0.6), leak=0.05, repeatable=False)
 
# Events per person (PostHog-style: page views, feature usage, flag exposures).
for i in range(400):
    person, t = f"person-{i}", i * H
    onboarded = i % 2 == 0
    db.insert(person, t, "action", {"event:onboarding_completed" if onboarded else "event:onboarding_skipped"})
    db.insert(person, t + 10 * M, "pageview", {"page:/pricing" if i % 3 == 0 else "page:/docs"})
    if onboarded and i % 4 == 0:
        db.observe(person, "converted", t + D)   # the outcome happened; no labelled cause
 
db.settle(400 * H + 30 * D)                       # windows that passed without it count against
print(db.link_report("converted", min_support=5))

Insert each person's events as they arrive (from a PostHog export or your own pipeline), call observe when an outcome happens, settle periodically, and link_report for strengths with 90% intervals and lift. top_causes(person, "churned", at=..., explain=True) explains one person's outcome.

With a connector (available on request)

The PostHog connector does this from a live PostHog project. A PostHog HTTP Webhook destination streams events to it, a HogQL backfill imports history so links start from your data, and PostHog events ($rageclick, $exception, conversions, cancellations) map to outcomes. Only allow-listed properties are kept. It produces the feature-effect report with confounder warnings, and explains one person's outcome. It is available on request: see Connectors for details and how to get access.

Explanation

• Soft credit, not labels. When an outcome is observed, each event in its kernel window gets credit in proportion to how much it explains it; when a window passes without the outcome, its events get evidence against. Over many users this estimates association strengths.
• Credible intervals, not just means. Beta counts give each link an interval; link_report shows it, so a strong-looking link with little data looks uncertain.
• Observational ≠ causal. Users who try a feature differ from users who don't. A strong link is a hypothesis; confirm it with a PostHog experiment (feature flag with a holdout) before acting on it.

Benchmarks

Both results below use synthetic data with known ground truth, so every recovered effect can be checked against the one we planted.

Core simulation (heapdb-core v0.4 tests, 3,000 users, noisy-OR generator): Spearman correlation between true and learned strengths 0.964 on seeds 7 and 11. Every real cause ranks above every null feature. A planted confounder z (no effect, co-occurs with the strongest cause 80% of the time) is only partly separated: on seed 7, strongest cause 0.165, moderate cause 0.134, z 0.127, nulls about 0.065. With uniform credit, z climbs above the moderate cause.

PostHog-format replay (PostHog connector, synthetic data; 2,000 users, 67,136 events):

Sources: core simulation from heapdb-core (tests/test_v04_observe.py, examples/product_analytics.py); PostHog-format replay from the PostHog connector, on synthetic data.