AI agent memory: recall what matters now

Overview

An agent accumulates observations, tool results, user preferences and facts over many turns and sessions. Stuffing all of it into the context is expensive and noisy, and plain similarity search returns things that match the words but went stale. Every turn asks the same question: of everything this agent has seen, which memories are most useful right now?

We built heapdb-memory to answer it. Every recall ranks candidate memories by relevance, recency since last use, importance, learned usefulness and rules, and explains each result in one line. When your agent reports which recalled memories it used (mark_used) and which it ignored (mark_ignored), HeapDB learns which kinds of memory help which kinds of question: dated memories for "when" questions, numbers for "how many", preferences over raw tool output.

Here's the second recall from examples/quickstart.py, copied from a real run. Ten days earlier the agent used the diet memory, and since then a newer diet memory has superseded it:

RECALL 'is Alice vegetarian? what does she eat?' IN user:alice (4 candidates)
  1. [69652055c595] Alice eats fish now (pescatarian)  score=0.1621
      rel 0.93 (bm25 0.93) x recency 1.00 (age 0.0d) x importance 1.10 x useful 0.16 [date:no=0.02, ent:no=0.02, kind:preference=0.02]
  2. [ccb036d88669] Alice's flight to Lisbon leaves Friday 9:40  score=0.0161
      rel 0.14 (bm25 0.14) x recency 0.90 (age 10.0d) x importance 0.95 x useful 0.14 [source:unknown=0.01, q:no=0.01, qt:what&date:yes=0.01]
  3. [4699ee7da1d8] Alice is vegetarian and allergic to peanuts  score=0.0159
      rel 1.00 (bm25 1.00) x recency 0.90 (age 10.0d, used 1x) x importance 1.04 x useful 0.17 [date:no=0.02, ent:no=0.02, kind:preference=0.02] x rules superseded x0.1

The old diet memory is still the best text match (rel 1.00), but the superseded rule (×0.1) puts it below its replacement. Memory ids are random per run.

How it maps

The score is a log-linear product; each exponent $w$ is configurable, and $w = 0$ switches that component off (design doc):

Design notes

• Recency to fit the workload. For recall over long histories where any point may be asked about, set w_recency=0 (the setting tuning picked on LoCoMo). For temporally local workloads, keep a recency weight.
• Feedback is stored as events. Each feedback call adds one small retrieval event to the HeapDB log, and forget(namespace=...) erases a namespace completely.
• Runs in your process. The memory layer is Python on the Rust heapdb engine, with in-memory indexes, JSONL persistence and exact vector search. Shape features and query types use English heuristics.

Setup

Python 3.10+. heapdb-memory 0.1 is on PyPI and pulls in heapdb 0.5 (the Rust engine, prebuilt wheels) as its only required dependency.

pip install "heapdb-memory[mcp]"
# extras: mcp, langgraph, llamaindex, embeddings (sentence-transformers), onnx (local ONNX encoder)

Python SDK

from heapdb_memory import Memory
 
mem = Memory("./memory", namespace="user:alice", autosave=True)   # dir is created / loaded
 
mem.add("Alice is vegetarian and allergic to peanuts", kind="preference", key="diet")
mem.add("Alice's flight to Lisbon leaves Friday 9:40", kind="fact", tags=["travel"])
mem.add("weather api: rain in Lisbon on Friday", kind="tool_result", source="tool")
 
res = mem.recall("plan Alice's Friday dinner", k=3)
print(res)                              # ranked, with explanations
for r in res:
    r.id, r.text, r.score, r.explain()  # also r.relevance, r.recency, r.importance, r.usefulness
 
mem.feedback(used=[res[0].id])          # used -> confirm; the other shown ones -> ignored
mem.add("Alice eats fish now", kind="preference", key="diet")   # supersedes the old diet memory
mem.forget(res[2].id)                   # or mem.forget(namespace="user:alice") to erase a user
print(mem.stats())
print(mem.explain_usefulness())         # which kinds of memory turned out useful

MCP server

heapdb-memory-mcp serves the tools remember, recall, mark_used, mark_ignored, forget and list_memories over stdio. Its instructions tell the model to recall before answering and to report mark_used afterwards.

Claude Code:

claude mcp add heapdb-memory -e HEAPDB_MEMORY_NAMESPACE=user:me -- heapdb-memory-mcp

Claude Desktop (claude_desktop_config.json) or Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "heapdb-memory": {
      "command": "heapdb-memory-mcp",
      "env": { "HEAPDB_MEMORY_DIR": "/Users/me/.heapdb-memory", "HEAPDB_MEMORY_NAMESPACE": "user:me" }
    }
  }
}

If the script is not on the client's PATH, use the absolute path of the venv's heapdb-memory-mcp.

Framework adapters

Install the langgraph or llamaindex extra. Both adapters are covered by the repository's tests (tests/test_adapters.py).

LangGraph: HeapDBStore is a BaseStore, and search(query=...) is a HeapDB recall.

from heapdb_memory.adapters.langgraph import HeapDBStore
store = HeapDBStore(Memory("./memory"))
graph = builder.compile(store=store)

LlamaIndex: a memory block for llama_index.core.memory.Memory.

from heapdb_memory.adapters.llamaindex import HeapDBMemoryBlock
block = HeapDBMemoryBlock(memory=Memory("./memory"), namespace="user:alice", k=5)

Explanation

How recall scores

1. Candidates: the top 50 memories by BM25, unioned with the top 50 by embedding cosine if an embedder is configured (nothing is downloaded implicitly).
2. Score: relevance × recency since last use × importance × learned usefulness × rules, as in the formula above. With all $w = 1$ it is HeapDB's own kernel × dose × links × rules score with relevance in front.
3. EXPLAIN: each result carries the BM25 and cosine parts, age, recency, importance, the strongest usefulness links and any rules applied, rendered as one line.

How learning works

mark_used inserts a text-free HeapDB retrieval event per memory, carrying the memory's shape features crossed with the query type, and confirms it for useful; mark_ignored rejects it. Usefulness depends on the question, not only the memory: a dated memory helps a "when" question and is noise for a "who" question, so HeapDB learns over (query type × memory shape). Each confirmed use also resets the memory's recency, so memories that keep paying off stay fresh.

Links are hierarchical: a new namespace with no feedback borrows what all the others have learned, and forget(namespace=...) erases a user including their share of the global links.

What it learned on LoCoMo

Global useful links after training on every conversation except conv-26 (prior mean 0.010), from RESULTS.md:

• Dates for "when" questions: qt:when&date:yes P(useful) 0.0300 (evidence n 1403), while qt:when&date:no is 0.0006.
• Numbers for "how many": qt:howmany&num:yes 0.0278 (n 261).
• Not turns that are themselves questions: q:yes 0.0006. A turn by the person the question names (qsrc:yes) is preferred over qsrc:no (0.0006).

These patterns carry over to conversations the model has never seen, which is where the held-out gain comes from.

Benchmarks

Learned usefulness improves retrieval on LoCoMo by +0.051 MRR over an equally tuned BM25 + MiniLM hybrid, and the gain holds with realistic noisy feedback: +0.035 with half the evidence citations missed, 10% false citations and only half the training questions giving feedback (noisy-feedback results). These numbers measure retrieval of evidence turns (Recall@k and MRR). The full methodology is in RESULTS.md.

Evidence retrieval on LoCoMo (Maharana et al., ACL 2024): 10 conversations, 5882 dialog turns as memories, 1982 questions with evidence. Learning is leave-one-conversation-out: usefulness is learned only from the other 9 conversations, with oracle feedback (for each training question, recall the top 10, mark_used the evidence turns among them, mark_ignored the rest). The held-out conversation never receives feedback. +tuned rows pick their weights by nested cross-validation on the training conversations. The tables are copied verbatim from the committed RESULTS.md, which also has per-category numbers, ablations, tuning choices and latency.

Recall@k = share of a question's evidence turns in the top k; MRR = 1/rank of the first evidence turn. Per-conv MRR = mean ± std over the 10 conversations (for +learned methods these are the 10 leave-one-conversation-out folds).

Paired difference in MRR vs bm25 (bootstrap over questions, 95% CI):

• Learned usefulness is the main gain: abl:bm25+learned vs bm25, ΔMRR +0.033 [+0.026, +0.039] on held-out conversations.
• At equal relevance and equal tuning: heapdb+emb+learned+tuned vs bm25+embedding+tuned (BM25 + MiniLM hybrid), ΔMRR +0.051 [+0.043, +0.058].
• Recency off for archive recall: LoCoMo asks about any point in a long history, so nested-CV tuning picked w_recency=0 in 18 of 20 fold choices (the default 30-day decay is -0.043 MRR [-0.052, -0.035] here, heapdb vs abl:bm25+importance). LoCoMo asks about any point in a 6-10 month history and never re-uses a memory before the test query, so it cannot test recency for temporally local questions or mark_used reinforcement.
• Tune the weights to the workload: with default weights, heapdb scores 0.421 vs 0.459 for BM25, and heapdb+learned is ΔMRR -0.005 [-0.014, +0.004]. The gains come from learning and a recency weight that fits the workload, which nested cross-validation picks.

Embedding model. sentence-transformers all-MiniLM-L6-v2, ONNX export, run with OnnxEmbedder. HuggingFace was not reachable in the evaluation environment, so the model files came from an npm redistribution of the same model (Apache-2.0); its vectors were checked against an independent fp16 safetensors copy through a separate NumPy BERT forward pass (cosine 1.0000 on test sentences) and against the published reference similarities of sentence-transformers.

Dataset license. LoCoMo is CC BY-NC 4.0. heapdb-memory does not redistribute it: the evaluation downloads it at run time and checks its sha256. Reproduce with python -m evals.locomo.run --onnx-model DIR && python -m evals.locomo.report (see the evaluation README).