Sign in
Docs Architecture

Storage, distribution and system architecture

A HeapDB node is a set of entity shards; each shard runs write, read and learn paths over the same data.

A HeapDB node is a set of entity shards; each shard runs three paths over the same data.

Events write down, questions read across, feedback learnsArchitecture of one entity shard in three rows. Write path: event streams flow into ingest and an append-only log, which seals time-bucketed segments with bounds, which feed hot heaps keyed by forward decay. Read path: a question (top k causes of E) goes to the planner (kernel window and bounds), then a bounded scan that stops at the k-th best, producing a ranked answer with EXPLAIN. Segments supply candidates to the bounded scan; hot heaps answer frequent effects directly. Learn path: feedback (confirm or reject) updates the link store of Beta counts, which supplies link strengths to the planner; the kernel registry supplies kernel values to the bounded scan. The ranked answer loops back to feedback when a person confirms the cause.
One entity shard. Write path across the top, read path through the middle, learning along the bottom.

Events flow into an append-only log and seal into time-bucketed segments; questions read across those segments with bounds from the link store and kernel registry; confirmed answers flow back into the link store.

Components

  • Log and segments. An LSM-style log seals segments by time. Old segments compact into coarser buckets that keep only their bound summaries and indexes.
  • Hot heaps. For effect types queried often ("errors" in ops), a per-entity forward-decay heap or tournament tree is maintained on insert, so the answer is a peek.
  • Sharding. Shards are keyed by entity (user, service, account), so most queries hit one shard. Cross-entity questions ("which service's deploy caused this incident?") run local top-k on each shard and merge with a k-way merge. They take a single effect type only, and merged scores are comparable only once calibrated; the merge does not renormalise per entity, because that would rank each entity's best guess as equally likely.
  • Deletes and corrections. Tombstones with lazy deletion: a heap entry is checked against the log when popped, never searched for in place.
  • Deployment. Single-user streams (health, personal finance) fit an embedded, on-device engine; ops and security fit a server deployment.
  • Forgetting. forget_entity drops an entity's events, segments, links and calibration records; see Privacy.