Getting started with heapdb-core

Install heapdb from PyPI, run the quickstart and the examples, and build from source to contribute.

heapdb-core is the HeapDB engine: a Rust engine with a Python API that you install with pip install heapdb. It runs inside your program with JSONL snapshots, or as a network service with heapdb-server. On 1M events, a top-causes query runs 16× faster at p50 than the same query in SQLite (benchmarks and methodology).

Install

pip install heapdb

Version 0.5 is on PyPI with prebuilt wheels (abi3, CPython 3.10+) for Linux x86_64 and aarch64, macOS (Intel and Apple silicon) and Windows x64. No Rust toolchain is needed and there are no runtime dependencies. The Python API is the same as the earlier pure-Python versions, and differential tests check that the Rust engine returns the same results. Installing from source instead (for example from a git checkout) needs a Rust toolchain; see Contributing.

Quickstart

from heapdb import HeapDB, LogNormalKernel
 
H = 3600
db = HeapDB(bucket_seconds=900)
db.register_effect(
    "bloating",
    kernel=LogNormalKernel(median=2 * H, sigma=0.6),   # peaks ~1.4h, long tail
    priors={"beans": (3, 1), "dairy": (2, 1)},          # Beta(alpha, beta)
)
 
db.insert(entity="me", time=8 * H,  type="meal", features={"oats", "dairy"})
lunch = db.insert(entity="me", time=13 * H, type="meal", features={"beans", "onion"}, weight=1.5)
 
res = db.top_causes(entity="me", effect="bloating", at=15 * H, k=3, explain=True)
for c in res:
    print(c.rank, c.event.short(), round(c.score, 3))
    print("   ", c.explanation)
print(res.stats)            # segments scanned / pruned
 
# Tell HeapDB what really happened. A confirmation splits credit across the
# event's features by their share of the score; rejected features get beta += 1.
db.feedback("me", "bloating", at_time=15 * H, confirmed_event_id=lunch.id, rejected_ids=[])
 
db.save("me.jsonl")
db2 = HeapDB.load("me.jsonl")

Run the examples

The examples in examples/ are plain scripts that need only the installed package. Clone the repository to read and run them; there is nothing to build:

pip install heapdb
git clone https://github.com/heapdb/heapdb-core.git
python heapdb-core/examples/bloating.py      # meals and a bloating effect, with feedback
python heapdb-core/examples/incident_rca.py  # a week of changes, then a latency spike
python -m heapdb --demo                      # interactive REPL

examples/incident_rca.py answers a week of noisy changes (234 events, 203 segments) by scoring 5 events, and checks the pruned answer against brute force. The demo on this site has a small preset based on it.

Query language

TOP <k> CAUSES OF <effect> FOR <entity>[,<entity>...] [AT <time>] [WITHIN <duration>] [EXPLAIN]
FEEDBACK <effect> FOR <entity> AT <time> CONFIRMED <id> [REJECTED <id>,<id>...]
INSERT <type> FOR <entity> [AT <time>] [FEATURES <f>,<f>...] [WEIGHT <dose>] [ATTRS k=v,...]
CALIBRATION OF <effect> [BINS <n>]

<time> can be seconds, now, now-2h or ISO-8601. <duration> can be 90, 30m, 12h or 2d. The REPL also has meta commands: .effect, .prior, .events, .save, .load, .help.

Repository layout

Path	What it holds
`crates/heapdb/`	The engine, a Rust library: events, lag kernels, Beta links + noisy-OR, time-bucketed segments, pruned top-k, explain, feedback, observe / settle, snapshots, query language
`crates/heapdb-cli/`	Native `heapdb` REPL and query-file runner
`bindings/python/`	PyO3 extension `heapdb._core` and the thin `heapdb` Python package (re-exports, type stubs, CLI I/O)
`tests/`	Python test suite, run against the built extension
`examples/`, `benchmarks/`	Python scripts that use the installed `heapdb` package

Contributing

We welcome contributions, issues and design discussion; start with CONTRIBUTING.md. Engine logic lives in Rust, and the Python package stays a thin binding with the same names and semantics.

Building from source needs a Rust toolchain (stable, 1.82 or newer, for example via rustup) and maturin:

git clone https://github.com/heapdb/heapdb-core.git
cd heapdb-core
python -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop --release   # build the Rust extension
pytest                      # Python test suite
cargo test --workspace      # Rust tests

pip install git+https://github.com/heapdb/heapdb-core also builds from source and needs the same Rust toolchain. Good first issues include new kernels, benchmarks and query-language features. Any change to the query engine must keep pruned top-k equal to brute force (tests/test_topk.py and crates/heapdb/tests/topk_props.rs).