Sign in
Docs Roadmap

Open problems, privacy and roadmap

What we're working on in learning and calibration, how privacy is built into the data model, and what comes next.

The most interesting work ahead is statistical: getting the most out of every piece of feedback. Here's what we're working on, how privacy is built in, and what comes next.

What we're working on

  • Separating correlated features. Link learning credits what co-occurs with an effect. Proportional credit and joint causes keep riders from being inflated; detecting partial overlap between features is next.
  • Strong starting priors. Good priors per effect type carry the first weeks of use. Shared global evidence helps, and curated priors per domain help more.
  • Ranking and causal proof. Feedback updates ranking weights; experiments and bisection prove a cause. See Design scope.
  • Calibration. A score of 0.61 should mean roughly 61% of such answers are confirmed. The calibration report measures this today; automatic recalibration is next.
  • Feedback coverage. People judge the answers they are shown, so judging the other top candidates too keeps the links balanced.

Privacy

Health and finance streams are sensitive, so privacy belongs in the data model, not only in deployment.

  • Forget an entity. forget_entity(entity) removes its events, segments, links and calibration records; scrub_global=True also subtracts what it contributed to the shared global layer. (Implemented.)
  • Per-entity encryption keys for data at rest, with deletion by destroying the key (crypto-shredding), which fits the tombstone model. (Planned.)
  • Audit log of reads and feedback per entity. (Planned.)
  • On-device by default for personal health streams. (Planned.)
  • Differential privacy for anything shared across entities, such as global link and kernel statistics. (Planned.)

Roadmap

Today HeapDB is an event-ranking engine: a Rust engine with a Python API (pip install heapdb, 0.5, in-process with JSONL snapshots), an agent-memory layer on it (pip install "heapdb-memory[mcp]", 0.1) and connectors, available on request. A full database is where we're headed. Our network server, heapdb-server, is on heapdb-core main with an edge container image (Self-hosting). An LSM-backed storage engine and sharding come after that.

  1. Reference implementation (Python, single node). Shipped as heapdb-core v0.1.
  2. Learning and confounding: proportional credit, hierarchical priors, joint causes, plausibility rules, tighter segment bounds, group queries, calibration report. Shipped as heapdb-core v0.2.
  3. Real feedback in one domain: code changes via the GitHub connector (available on request), personal health next. Reverts and "Caused by #123" confirm, postmortems reject; this stress-tests priors, confounding handling (ai_authored) and explanation quality on real repositories.
  4. Pre-merge risk (feature_risk, rank_candidates, RISK OF): rank open PRs by learned risk before they merge. Shipped as heapdb-core v0.3.
  5. More connectors: Sentry, PagerDuty, Datadog, GitLab and Linear, on the same normalized records and bridge.
    • Community connectors repository (planned): an open-source home for connectors, so you can write and share your own.
  6. Benchmarks: reproducible benchmarks/ in heapdb-core against brute-force Python, SQLite and PostgreSQL. Published. Rerun for the Rust engine: faster than SQLite and PostgreSQL at every size measured, 16× faster than SQLite at p50 on 1M events (Benchmarks). Next: real (non-synthetic) streams.
  7. Agent memory: explained recall that learns which memories an agent uses. Released: heapdb-memory 0.1 on PyPI, with LoCoMo results. Learned usefulness holds its gain with realistic noisy feedback (results). Next: a benchmark that tests recency and reinforcement.
  8. Calibration correction: fit score → probability per effect from recorded feedback.
  9. Kernel learning: fit lag distributions from confirmed feedback.
  10. Embedded engine (Rust). Shipped as heapdb 0.5.0 on PyPI: the same Python API with prebuilt wheels, identical results to the Python engine. Next: an LSM-backed segment store and per-entity encryption.
  11. Hosted service (early access): HeapDB Cloud, with a managed data plane and a web console. Its Python client is heapdb.connect(), in heapdb 0.6.0 and later.
  12. Distributed mode: entity sharding and cross-shard top-k merge on calibrated scores.

Your use cases decide which of these we build first. Share yours on the board.