Docs Data model

Data model

HeapDB stores four things — events, effects, links and kernels — and scores each event against a question.

HeapDB stores four things: events, effects, links and kernels. Only events are high-volume; the rest are small, per effect type.

ConceptShapeExample
Event(entity, time, type, features, weight), append-only(me, 19:30, meal, {dairy, onion, beans}, 1.5)
Effect(entity, effect type, time), the question(me, bloating, 22:45)
LinkBeta(α, β) per (feature, effect type)beans → bloating: Beta(8, 3), strength 0.73
KernelLag density per effect type, with a support windowbloating: log-normal, median 2.5 h, support 12 h

The score

The score of event e for effect E at time t:

score(e∣E,t)=KE(t−te)⋅we⋅(1−∏f∈Fe(1−sf,E))\mathrm{score}(e \mid E, t) = K_E(t - t_e) \cdot w_e \cdot \Big(1 - \prod_{f \in F_e} \big(1 - s_{f,E}\big)\Big)

K is the lag kernel (zero for negative lag), w the event weight, and the bracket a noisy-OR over the event's features: any one strong feature is enough to implicate the event, and several weak ones add up. Link strength s is the Beta mean α / (α + β).

Core algorithms shows how HeapDB avoids computing this score for every event on every query.

Weight is dose

The weight w is the magnitude of exposure: portion size, lines changed in a deploy, transaction amount. It defaults to 1.0 and must be positive. It is not a measure of how likely the event is to cause anything; that propensity is what links learn from feedback, and putting it in the weight as well would count it twice.

Weights can be set explicitly or derived per event type from the event's attributes:

db.register_event_type("meal", weight=lambda a: a.get("portion", 1.0))
db.register_event_type("deploy", weight=lambda a: math.log1p(a["lines_changed"]))
db.insert(entity="checkout", time=t, type="deploy", features={"db_pool"}, attrs={"lines_changed": 420})

Final scores also pass through any plausibility rules for the effect, which can only lower them.