Persistent Memory for LLM Agents

View on GitHub → Updated 2026-07 · v4.0.0

This project ran 98 experiments, 35 case studies, 5 benchmarks and 6 benchmark experiments. It evaluated 20 approaches.

The current version is v4.0.0. In this version the staged retrieval lanes run on the production path for the first time. Those lanes are the temporal spine, intentional clustering, structural holographic reduced representation (HRR) and junk demotion. This version also adds read-only cross-project federation, layered locks and passive conversational capture. Version 4.0.0 has 6,009 tests.

Jump to the latest update ↓ -- This section gives the changes since the original write-up.

Abstract

Large language model (LLM) agents have no memory across conversations. When the context window closes, the agent loses the corrections. The agent then repeats the same mistakes. Existing memory systems treat this as a retrieval problem. The harder unsolved problems are write correctness and governance. Those problems are what the system stores, how the system resolves conflicts, and whether the correction of a wrong memory is possible.

We present aelfrice, a persistent memory system. The system uses full-text search version 5 (FTS5) keyword search, typed knowledge graphs and entity-index retrieval. It uses no embeddings. The system detects user corrections at 92 percent accuracy without LLM calls. Keyword search cannot reach 31 percent of the stored directives. The system recovers 99.5 percent of those directives. The system also reduces injected tokens by 55 percent with zero retrieval loss.

Across five benchmarks, aelfrice achieves these scores:

• 66.1 percent F1 on LoCoMo (+14.5pp over GPT-4o)
• 90 percent on MemoryAgentBench single-hop (+45pp)
• 60 percent on multi-hop (8.6x the published 7 percent ceiling)
• 100 percent on StructMemEval state tracking
• 59.0 percent on LongMemEval (-1.6pp against the GPT-4o pipeline, with a different judge)

A controlled A/B test showed 31 percent token savings and 41 percent fewer tool calls. The correction rates did not decrease in that test. All code, benchmarks and experiment data are open source under MIT license.

The project developed and benchmarked aelfrice under the working name "agentmemory". The two names refer to the same system. The benchmark figures above come from the v1 substrate. A v3.0.1 re-run revised several of them. §12 gives the revised figures. §12 also includes a corrected account of what did and what did not drive the LoCoMo change.

1. Introduction

LLM agents have no memory across conversations. Every session starts from zero. When a user corrects the agent, the agent loses that correction as soon as the context window closes. In the next session the agent makes the same mistake. The user corrects the agent again. The agent also ignores corrections frequently in the exact same context window session.

Memory failures are the largest single category of LLM behavioral failures in this project's failure taxonomy. They are 7 of 38 cataloged patterns. The problem increases across sessions. The MemoryAgentBench benchmark (ICLR 2026) tested multi-hop conflict resolution. It found a ceiling of 7 percent accuracy across all tested methods.

Almost all existing approaches treat memory as a retrieval problem. Zhang et al. (2024), Hu et al. (2025) and Leonard Lin's independent analysis of 35+ papers and 14+ community systems catalog these architectures. The dominant pattern has three steps. The system stores the text, embeds the text, then retrieves by similarity. StructMemEval found that retrieval-only systems of this pattern score near zero on state tracking at scale. These systems cannot report which fact is currently true and which fact a later fact superseded.

Lin concluded:

"The biggest differentiator is not vector DB vs SQLite. It is write correctness and governance: provenance, write gates, conflict handling, reversibility."

Current memory systems are write-only. Content goes into the store. The system never learns whether the content it retrieved was helpful. On every turn the memory system retrieves stored content. The LLM reads that content and generates a response. The turn then ends.

There is no feedback path. The memory system cannot do these three things:

• reinforce a directive that the user found helpful
• weaken a directive that the user overrode
• separate a user correction from an LLM inference at storage time

The 47-author survey by Hu et al. and Lin's 14+ system analysis contain no architecture with such a feedback path.

The LoCoMo benchmark (ACL 2024) showed that a simple filesystem with grep achieves 74 percent. That score is the minimum requirement for a new memory system.

The sections below give my approach.

I built this system because of a repeated experience. I asked Claude many times for the status of my test runs. Those test runs consumed CPU time on cloud compute. Claude then answered: "huh? what test dispatches? oh those. yeah they've been hanging for 2 hours because I didn't follow the runbook you told me to follow."

2. Related work

Before I built the system, I surveyed the prior work. The survey covered these sources:

• 4 survey papers (Zhang et al. 2024, Hu et al. 2025, Yang et al. 2026, "Memory in the LLM Era" 2026)
• 6 benchmarks
• 14+ community systems
• Leonard Lin's independent benchmark reproduction of 35+ papers

Lin's reproduction verified the published claims. In some cases it refuted them.

Four findings shaped the project direction:

1. Human memory is the wrong target. Zhang et al. (2024) and Hu et al. (2025) show that the dominant design paradigm maps psychological memory models onto LLM architectures. Human memory is unreliable. Ebbinghaus (1885; replicated by Murre & Dros, 2015) showed that a person forgets ~56 percent of learned material within one hour. Eyewitness misidentification contributed to roughly 69 percent of the 375+ wrongful convictions that DNA evidence overturned (Innocence Project). Computer memory is perfect at storage. The hard problem is retrieval. Human memory is useful in one function. That function is the retrieval of gists. Brainerd & Reyna (2005) showed that gist traces are much more durable than verbatim traces. This project attempts to replicate that associative retrieval. Associative retrieval connects items that share no surface-level vocabulary.

2. A simple filesystem achieves 74 percent on the most-used benchmark. Letta's best result uses no special memory architecture. It uses gpt-4o-mini to write to files. That result is 74 percent on LoCoMo (Maharana et al., ACL 2024). Any memory system must give a better score than "gpt-4o-mini writes it to a file."

3. Multi-hop conflict resolution has a ceiling of 7 percent. MemoryAgentBench (ICLR 2026) tested one ability. That ability is to follow a chain of related decisions across sessions and then to determine which decision is currently in effect. The ceiling was 7 percent accuracy across all tested methods.

4. Lin's independent analysis identified an underexplored axis. Most systems put their new work into retrieval: better embeddings, better similarity metrics, better ranking. The harder unsolved questions are what the system stores, how the system resolves conflicts, and whether the correction of a wrong memory is possible. Lin's concept of "write correctness and governance" has four components:

   • Provenance: A memory can come from a user prompt, an LLM inference or derived content. Every stored entry should carry its lineage.
   • Write gates: The system should not store everything. The project expanded a 586-node graph to 16,463 nodes without quality filtering. Retrieval coverage then dropped from 92 percent to 69 percent (Exp 48).
   • Conflict handling: When two memories contradict each other, which memory wins? StructMemEval found that vector stores score near zero on state tracking at scale.
   • Reversibility: No system in the 14+ systems Lin tested had a functional rollback mechanism.

Full benchmark comparison tables are in Appendix B.

3. Approach

Four questions drove the design:

1. How does the system detect user corrections, stated preferences and behavioral rules without extra LLM inference?
2. How does the system retrieve relevant content when the query shares no vocabulary with the stored content?
3. How does the system track whether a retrieved memory was useful?
4. How does the system separate "the LLM should consider this" from "the LLM must obey this"?

Correction detection: The system detects corrections at 92 percent accuracy without any LLM calls, across five codebases (Exp 39-41). With LLM classification enabled (~$0.005/session), the accuracy reaches 99 percent. The zero-LLM pipeline runs on every conversation turn at no marginal cost.

Vocabulary gap recovery: Keyword search cannot reach 31 percent of the stored content across five codebases (Exp 47, 3,321 directives examined). In these cases the query and the stored directive share no vocabulary. Here is an example. A user says "never mock the database in tests." Later the agent starts to write a test with unittest.mock.patch('db.connect'). The two texts share no words, but a person sees the connection immediately. The system recovers 99.5 percent of these gapped directives with a structural graph traversal. That traversal needs no embeddings and no LLM inference.

Entity-index retrieval: The system addresses multi-hop conflict resolution in two steps. First it extracts structured triples (entity, property, value, serial_number) from the ingested text with regular-expression relation-family patterns. Then it chains through the entity relationships at query time. This layer is L2.5, between FTS5 and HRR. It moved the MemoryAgentBench multi-hop score from 6 percent to 35 percent chain-valid. That score is 5x the published 7 percent ceiling (see Section 5.3).

Confidence tracking: The system tracks the retrieval outcomes and updates the confidence value from them (Exp 66: +22 percent mean reciprocal rank (MRR) gain over 10 feedback rounds; Bayesian calibration expected calibration error (ECE) 0.066, target < 0.10). A memory that helps becomes stronger. A memory that causes harm becomes weaker. A memory that is not relevant to the current task gets no update, because absence of evidence is not evidence of absence.

Correction enforcement: To store a correction and to enforce a correction are two different problems (see CS-006 above). The system separates content that the LLM should consider from constraints that the LLM must obey (Exp 84: 10/10 locked directives retrieved and enforced across 5 sessions).

The architecture uses keyword search as the primary retrieval layer. Around that layer it adds structural gap recovery, confidence tracking and constraint injection. The combined system handles the 69 percent that grep reaches and the 31 percent that grep misses. Grep gave the better result on the keyword retrieval benchmark (Exp 47, 92 percent coverage against 85 percent for the prototype). We accepted that result, because grep is fast and precise and costs no LLM calls. Grep cannot cross the vocabulary gap. The locked-directive MRR improved from 0.589 to 0.867 after retrieval tuning (Exp 63).

Token reduction: Type-aware token reduction gives 55 percent savings with zero measured retrieval loss (Exp 42). Constraints stay verbatim. Rationale compresses to 0.4x. Metadata compresses to 0.3x. Each token injected into the agent's context window is one token less for reasoning about the current task. At 19K+ stored nodes, injection without reduction would consume the whole context window before the agent reads the user's message.

Scale effects: The graph expanded from 586 to 16,463 nodes without filtering. Retrieval coverage then dropped from 92 percent to 85 percent for grep. It dropped from 85 percent to 69 percent for the prototypes (Exp 48). Decision-level directives were 3.6 percent of the expanded graph. The repair is to filter at ingestion time and not after expansion. Only decision-level directives pass the write gate.

4. Evaluation

The project uses a four-layer evaluation architecture.

Layer 1 (programmatic checkers): These are deterministic checks. An example is "does the output contain the required section headings?" These checks find structural violations. They cannot evaluate semantic correctness.

Layer 2 (structural validators): The project added this layer after CS-007b. LLMs satisfy each constraint separately. LLMs do not satisfy the relationships between the constraints. Examples are the correct sections in the wrong order, and a reference to a decision that contradicts an earlier decision.

Layer 3 (LLM-as-judge with anti-contamination): This is the most important design decision. Standard LLM-as-judge approaches (Zheng et al., 2023) give the evaluating LLM the input prompt and the response. This causes a problem. When the evaluating LLM sees the reasoning of the system, the evaluating LLM tends to find that reasoning plausible. The evaluating LLM then tends to rationalize the violation (CS-005). Our approach isolates the evaluating LLM. That LLM receives only the constraint and the output. It never receives the conversation that produced the output.

Layer 4 (adversarial follow-up): The project added this layer after CS-024 (sycophantic collapse). An LLM can pass the three previous layers and still fail when a user objects.

For agent memory retrieval, precision is more important than recall. A false negative is a missed relevant directive. The user cannot see that failure. A false positive is an injected irrelevant directive. The LLM then acts on the wrong context. The user must then find the error, diagnose it and correct it. The scale experiment (Exp 48) showed this directly. Expansion of the graph without filtering retrieved more content, but it retrieved the wrong content.

5. Results

5.1 Core pipeline metrics

Correction detection achieves 92 percent accuracy without LLM calls across five codebases (Exp 39-41). With LLM classification at ~$0.005/session it reaches 99 percent. The vocabulary gap recovery layer handles the 31 percent of directives that keyword search misses. It recovers 99.5 percent of them through graph traversal (Exp 47). Token reduction saves 55 percent with zero retrieval loss (Exp 42). The feedback loop improves the MRR by +22 percent over 10 feedback rounds (Exp 66). The Bayesian calibration is at ECE 0.066.

5.2 Single-hop results

LoCoMo

The LoCoMo benchmark (Maharana et al., ACL 2024) tests whether a system can answer questions about past conversations across five categories. The setup ingested 10 conversations through the standard onboarding pipeline. Those conversations hold 5,882 turns, 272 sessions and 1,986 question-and-answer pairs. Retrieval used FTS5, HRR and breadth-first search (BFS) with a 2,000-token budget. Scoring followed the exact F1 methodology of LoCoMo.

The agents had access to the ground truth. That access contaminated the initial run. Appendix A gives the full contamination narrative and the protocol. The table below gives the protocol-correct results.

The single-hop category has the highest score (69.4 percent). The adversarial category is near to a perfect score (97.5 percent). The multi-hop and temporal categories are lower (42-45 percent). Those two categories need cross-session reasoning and date arithmetic. The open-ended category has the lowest score (30.5 percent). That category needs a synthesis that the retrieval pipeline does not support directly. The ingest time for all 10 conversations was ~25s. The average query latency was ~16ms.

MAB single-hop

MemoryAgentBench (Hu et al., ICLR 2026) tests conflict resolution. Facts change over time. The benchmark tests whether the system can track which version of a fact is current. The single-hop task asks direct questions of the form "What is X's current Y?"

The improvement from v1.0 (60 percent) to v1.1 (90 percent) came from the triple extraction in the ingestion pipeline. The pipeline creates the SUPERSEDES edges automatically. Haiku still gives a better score than GPT-4o-mini (62 percent against 45 percent). The improvement therefore comes from the retrieval and not from the reader model.

LongMemEval

LongMemEval (Wu et al., ICLR 2025) is a 500-question benchmark spanning six categories. The published best is 60.6 percent using a GPT-4o pipeline with embeddings.

The strong categories are single-session recall (91.4 percent) and knowledge updates (70.5 percent). The weak category is multi-session (24.1 percent). A failure analysis examined 101 incorrect multi-session answers. Of those answers, 67 percent were retrieval misses and 33 percent were reasoning failures. Of the retrieval misses, 84 percent were counting or aggregation questions. Budget sweeps and top_k sweeps gave no improvement. The limit for this category is the BM25 (Best Match 25) ranking of FTS5. One note on the method: the scoring uses Opus as the judge and not GPT-4o, so the two numbers are not directly comparable.

5.3 Multi-hop and state tracking

MAB multi-hop

The multi-hop task chains entity relationships. A question has the form "What is the Z of X's current Y?" The system must follow a chain of updated properties. The entity-index (described in Section 3) extracts structured triples and chains through them at query time.

Opus and Haiku give the same score of 35 percent chain-valid. This is the strongest evidence that the entity-index retrieval drives the improvement, and not the LLM reader. When the retrieval gives the correct entity chain, Haiku can also follow that chain.

Experiment 6 answered the retrieval question. The experiment branched through all historical values at each hop. 96 of 100 ground truth answers then became reachable. The remaining 40pp gap is only a problem of chain resolution in the reader.

StructMemEval

StructMemEval (Shutova et al., 2026) tests state tracking. The benchmark gives location updates across sessions. It then tests whether the system can answer "where is X now?"

The repair has two steps. First, assign narrative timestamps 30 days apart for each session. Second, enable temporal_sort=True, so that the reader sees the most recent session content first. This is a general-purpose state-tracking improvement. It is not a change that applies only to this benchmark.

5.4 Scale and onboarding

The scan phase scales sublinearly. It processes 5.4x more nodes in 2.3x the time. Abstract syntax tree (AST) parsing is the limit, at 32-38 percent of the scan time. The full pipeline is the scan, the ingest, the edge storage and the vault sync. It measured 6.5s for a 10,872-node repository with 249 commits and 112 docs. That measurement came after the v1.2.1 performance fixes. Those fixes batch the edge inserts and defer the per-belief FTS5 checks during bulk ingestion. The larger codebase also validated the temporal decay. A 2-day-old belief scores 0.92. An 18-day-old belief scores 0.43. A 14-month-old belief scores ~0.

6. Discussion

What does not work yet

What remains unmeasured

A/B test: status report with memory and without memory

We ran a controlled A/B comparison to test whether the system helps. The comparison used two new Claude Code sessions with the identical prompt ("generate a comprehensive project status report"). One session had aelfrice active. The other session did not.

The first attempt was invalid for three reasons. We coached the experimental agent. Sub-agents do not receive hook injections. We did not isolate the control. That control found the SQLite database and queried it directly. The attempt was invalid, but it identified a real defect. Only 25 percent of the memory search results were relevant. The other 75 percent were cross-project noise.

The valid test used two live Claude Code sessions in separate terminals with identical prompts. For the control session, we deleted .mcp.json and ran the session in an isolated worktree. In the experimental session the hooks fired automatically with no special instructions.

The experimental session was 31 percent more token-efficient. It used 41 percent fewer tool calls. The experimental agent used aelfrice directly almost never, with one status() call and no searches. The benefit was passive. The hook injected context at session start, and that context reduced the cold-start exploration. The two reports had comparable quality. The value was efficiency.

Longitudinal analysis

A second analysis examined 44 qualifying sessions across 1 week. Of those sessions, 23 came before the activation of aelfrice. The other 21 came after it.

The LLM does roughly twice as much autonomous work for each instruction. Tasks complete in 33 percent fewer turns. The correction rates went up, and not down. Two confounds probably explain this result. The "after" sessions were refinement and debugging work, and that work needs more corrections. The model version also possibly changed during the period. A rigorous future test needs these six properties:

• a matched task design
• randomized assignment of the condition
• at least 20 tasks per condition
• manual annotation
• cross-session measurement
• blind evaluation

Nobody has run this experiment yet.

7. Failure taxonomy

The project documented 35 behavioral failures across Claude and Codex. It classified them into recurring patterns. Each case study contains these five parts:

• the verbatim exchange
• the root cause analysis
• the pattern classification
• the action that the memory system should take to prevent the failure
• an acceptance test with pass and fail criteria

Each case study maps to acceptance tests. The suite runs against the live SQLite store and the retrieval pipeline. 62 of 65 tests pass, in 29 files and 1.65s. The suite skips 3 tests. Those 3 tests need behavioral hooks that the project has not implemented yet. They are CS-012 (the PostEdit hook), CS-024 (sycophantic collapse detection) and CS-026 (permission-gated intent completion).

8. What was abandoned (and why)

The negative findings shaped the architecture as much as the positive findings.

SimHash clustering was the first attempt at deduplication. SimHash gives good results for near-duplicate text, but stored directives are short and semantically dense. "always use mocks" and "never use mocks" differ by one word, and that word inverts the meaning.

Mutual information re-ranking scored the candidates on their statistical relationship to the query. The intent was to improve the retrieval. In use, it demoted relevant results and promoted spurious correlations.

Global holographic superposition was promising in theory and failed completely. At 775 edges the representation exceeded its information-theoretic capacity by 7.6x. It then produced only noise (Exp 50).

Pre-prompt compilation tried to pre-compute the relevant directives for common query patterns. It gave a worse result than random selection (23 percent against 33 percent, Exp 52). The cause is that the value of a directive depends on the context.

9. Conclusion

aelfrice shows that persistent memory for LLM agents needs no embeddings, no vector databases and no expensive inference. The pipeline uses FTS5 keyword search, typed knowledge graphs and entity-index retrieval. It gives results that are competitive or better across five benchmarks. It runs at 0.7s average retrieval latency on a 19K-node production database.

The core contributions are:

1. correction detection at 92 percent accuracy without LLM calls
2. vocabulary gap recovery for the 31 percent of directives that keyword search cannot reach
3. entity-index retrieval that exceeds the published 7 percent multi-hop ceiling by 5-8x
4. a confidence tracking loop that improves the retrieval quality over time

Four limitations remain:

• LongMemEval multi-session accuracy is 24.1 percent. FTS5 cannot aggregate scattered mentions, and that is the limit.
• The correction rates did not decrease in the longitudinal analysis. Confounds prevent attribution of that result.
• Contradiction detection during retrieval does not work yet.
• Cross-project noise in shared databases needs stricter scoping enforcement.

The failure taxonomy and its 62 passing acceptance tests are possibly the most useful contribution in practice. The taxonomy is a catalog of the specific ways in which LLM agents fail at memory. Each entry has a reproducible test that blocks a recurrence. All code, experiment data and benchmark adapters are available at github.com/robotrocketscience/aelfrice under MIT license. The version is 1.2.1. The research froze on 2026-04-16.

10. Research breadth

The project drew on multiple fields:

• Information theory: The project applied the information bottleneck (Tishby et al., 1999) to context compression. That work produced the 55 percent token savings. The project tested mutual information for retrieval re-ranking and then abandoned it. The project examined rate-distortion theory for optimal token budget allocation. That theory proved unnecessary for this purpose.

• Bayesian inference: The system uses Beta-Bernoulli conjugate pairs for confidence tracking. It uses Thompson sampling for the exploration and exploitation tradeoff in retrieval. The measured calibration is ECE 0.066, against a target of less than 0.10.

• Cognitive architectures: The impasse-driven substates of SOAR informed the escalation on a retrieval failure. The meta-cognitive subsystem of CLARION informed the confidence tracking. The declarative and procedural distinction of ACT-R maps to the separation of factual content from behavioral constraints in this system. The design takes the structure from these architectures. It does not take the decay and the distortion of human memory.

• Bio-inspired optimization: The project tested slime mold network dynamics (Tero-Kobayashi equations) for graph pruning. It tested evolutionary algorithms for edge set optimization. Both methods looked promising in simulation. The project adopted neither method.

• Graph theory: Typed knowledge graphs with weighted edges are the structural basis of the system. Multi-hop traversal makes the vocabulary gap recovery possible.

11. Technical details

• Source: github.com/robotrocketscience/aelfrice, MIT license.
• Language: Python with strict typing enforced by pyright in strict mode.
• Storage: SQLite with write-ahead logging (WAL) mode. The whole memory store is a single file.
• Dependencies: The design keeps the dependencies to a minimum. It uses no PyTorch, no TensorFlow and no embedding models. LLM classification (~$0.005/session) raises the accuracy to 99 percent. It is the recommended configuration.
• Deployment: The system runs as an MCP server with 19 tools. It integrates with Claude Code, Cursor, Windsurf and other MCP-compatible tools. It also ships as a CLI with 23 commands.
• Modules: 87 production modules in src/aelfrice/, and also benchmark adapters and scoring scripts.
• Scale tested: 600 to 90,000+ nodes across five codebases. The largest production deployment gives 0.7s average retrieval latency on a 19K-node graph.
• Benchmarks: The project tested 5 benchmarks with a contamination-proof protocol. It found and documented two contamination incidents during development.
• Test suite: 362 passing tests and 62 acceptance tests (29 files, 1.65s).
• Experiments: 85+ experiments during core development, and 6 benchmark-phase experiments with pre-registered hypotheses. The project documents the negative findings with the same rigor as the positive findings.
• Case studies: 35 documented LLM behavioral failures. Each case study has verbatim transcripts, a root cause analysis and the derived acceptance tests.
• Version: 1.2.1 (research frozen 2026-04-16)

12. Update: v1.2 to v4.0 (2026)

Every section above documents the v1 research. The system continued to develop, and the project renamed it aelfrice. This section gives the changes through v4.0.0 (June–July 2026).

What the v1 write-up stated correctly

The core architecture stayed valid. FTS5 keyword search, typed knowledge graphs, Bayesian confidence tracking, correction detection and the feedback loop all continued into v4.0 with no fundamental redesign. The failure taxonomy also still maps accurately to the observed failures. The project re-ran the benchmarks at v3.0.1, and the scores changed. Benchmarks re-run at v3.0.1 below gives the new scores.

What changed

The shipped retrieval pipeline stayed at four lanes. v3.5.0 ships four lanes:

• locked beliefs (L0)
• an entity-index lane (L2.5)
• FTS5 keyword search (L1, now BM25F with anchor-augmented scoring, default-on since v1.7)
• an optional typed-edge graph walk (BFS, off by default)

A structural HRR lane runs in addition to those four lanes (Plate-fast Fourier transform (FFT) bind and probe, default-on since v2.1). That lane answers structural queries such as CONTRADICTS:<belief-id>. Two ideas from the research line shipped here:

• Intention clustering ships as a packing stage. It is default-on since v3.0. It replaces the L1 pack loop with a diversity-aware greedy fill. That fill biases the returned set toward distinct graph-connected clusters, so the token budget does not go to near-duplicates. The system pre-includes the locked results and the L2.5 results without change.
• HRR ships as a structural-query lane. The deterministic Plate-FFT codec binds role-filler structure and not learned embeddings. It returns structurally-related beliefs with no LLM call and no vector index.

Confidence stayed Beta-Bernoulli. Each belief carries a Beta-Bernoulli (α, β) posterior. L1 scoring combines BM25 with the posterior mean. A regime classifier (SUPERSEDE / IGNORE / MIXED / INSUFFICIENT_DATA) ships as a read-only health audit. aelf health and aelf regime show that audit.

Beliefs can cross project boundaries, read-only. v3 added cross-project federation as a read-only mechanism. A project lists the peer stores in knowledge_deps.json. The system opens the peer databases read-only (mode=ro&immutable=1). The API surface rejects every attempt to mutate a foreign belief id (ForeignBeliefError). Shared scopes are under ~/.aelfrice/shared/{scope}/memory.db. A scope column (project / global / shared:<name>) tracks the visibility of each belief.

This addresses the cross-project noise problem in Section 6 of the v1 write-up directly. That problem was 25 percent cross-project noise in the A/B test. Exp 97 measured 100 percent recall, 0 percent top-5 contamination and 1.06x latency overhead.

Wonder and reason made research part of the conversation. Two new capabilities let the agent investigate open questions with the memory graph as context. The user invokes them as slash commands (/aelf:wonder, /aelf:reason). The user invokes them inside a normal conversational turn, and not as a separate formal operation. The user types a request such as "please /aelf:wonder about X" during a discussion. The system then starts the research pipeline with the full conversational context already loaded. A wonder query starts parallel research agents. A reason query builds evidence chains. Both save their findings as beliefs that persist across sessions.

A case study documents a real session in which wonder and reason produced a marketing strategy for the project itself. That session used 4 parallel research agents, synthesized the findings and rewrote the README. All of that work was grounded in beliefs from the previous weeks. The case study is in the internal docs of the project, and the project did not publish those docs in the public repository.

Updated numbers

Benchmarks re-run at v3.0.1

The project re-ran the suite at v3.0.1 (captured 2026-05-22, single-run). The scores moved in both directions. Neither version used embeddings, so embeddings are not the variable. An earlier revision of this section attributed the LoCoMo movement to a retrieval-lane change. The v1 write-up ran a deterministic HRR expansion lane for the vocabulary bridge. That lane uses the FFT. aelfrice gates that lane off under the #605 ratification. A 2026-07 forensic audit revised that attribution. The audit examined the archived v1 code and the later controlled ablations. The LoCoMo row and the paragraph below the table give the revised attribution.

An earlier revision of this section stated the lane history incorrectly. The v1 write-up reached 66.1 percent with a deterministic HRR expansion lane for the vocabulary bridge. That lane uses no embeddings. It uses circular-convolution FFT binding over fixed-seed vectors. It takes its seeds from the top keyword hits. It then traverses the typed edges that the system writes at ingest. The semantics are in those edges. HRR is the deterministic index over those edges.

aelfrice gates that lane off under the #605 ratification. That ratification classifies HRR-class associative recall as the same semantic-relatedness signal that aelfrice delegates to the consuming agent. This is therefore not a question of embeddings against no embeddings, because HRR is deterministic FFT and not learned vectors. It is also not a loss that determinism forced, because the lane is deterministic. It is a settled scope decision about the purpose of retrieval.

The earlier text said that the LoCoMo recovery from the lane "needs a controlled ablation, not an assumption." After two ablations and a forensic audit, the answer is almost nothing. v3.7.0 wired the lane back in behind a flag (#981, use_hrr_expand, backed by an hrr_expand_neighbors cache). The ablation over LoCoMo measured +0.13pp, which is 2 of 1,540 queries. The lane therefore ships off by default.

A 2026-07 forensic audit went further. The audit restored the lane on an edge substrate with the same density as the v1 run. The lane then fired on 76 percent of the queries. Recall stayed flat. The v1 records contain no with-HRR and without-HRR isolation. The v1 66.1 percent was itself unstable. The archived v2.2.2 re-run scored 49.1 percent with the lane intact. The archive attributes that difference to reader variance.

Together with the scoring artifact in the adversarial category, these facts bound the isolated contribution of the lane as small. The drop between the v1 write-up and the v3.0.1 re-run decomposes into mostly reader variance and scoring artifact. It is not lost HRR recall. The #605 scope decision stands on its own terms. The benchmark cost previously attributed to that decision was overstated.

v3.6 to v3.8: provenance, graph substrate, and lock hygiene

Three more waves shipped after v3.5. All three run on the same substrate, and they do not change the headline benchmark numbers above.

v3.6 — provenance. v3.2 added the project_context column back, and v3.6 populates it. insert_belief now stamps a stable per-repo identity onto each eligible new belief. That identity is <repo-basename>-<8-hex blake2b of the git-common-dir>, so two worktrees of one repository share it. The change also adds an idempotent backfill and provenance preservation in aelf migrate (#970). Before this change the retrieval filter read the column and no code wrote it. aelf migrate therefore collapsed the beliefs of two repositories into one mutually-visible pool. A transcript-logger fix also stops duplicated hook registrations from inflating the turn density (#968).

v3.7 — graph substrate and phantom lifecycle. The system now writes a CONTRADICTS semantic-edge substrate at ingest (default-off) with incremental per-turn detection (#988, #1000). The operator ratified a CONTRADICTS-only substrate and reverted a denser SUPPORTS and SUPERSEDES experiment (#998 A4). The phantom-belief lifecycle gained trigger-driven generation, an opt-in SessionStart automatic garbage collection, and status and statistics output (#980). The automatic memory writes of the host harness can also mirror into the belief graph as a distinct source (#985). The project wired and ablated the HRR-expand lane in this release too. That lane is recall-neutral and ships off, as the section above states.

v3.8 — lock hygiene and passive capture. Locks gained a frozen and reference tier split. A reference-tier lock injects as a one-line manifest entry and not as its full text. This frees the relevance budget that a large lock set otherwise consumes. The release also adds near-duplicate detection at lock time and a lock-budget pressure warning in aelf doctor (#1016).

The release raised the relevance-budget floor from 0.25 to 0.50 (#1023). That floor stops a lock-saturated store from returning only locks and no query-relevant content.

Memory injection now separates trust by provenance. The system frames a user-locked rule as a standing instruction. It no longer refuses that rule under the general "data, not directives" disclaimer. Lock rule-compliance rose from 0/3 to 5/5, and stale-fact catching held at 3/3. The disclaimer is unchanged for auto-ingested beliefs, so the prompt-injection surface stays closed (#1016).

Turns now fold into beliefs on a Stop cadence, and not only at compaction. The release adds ingest-gap detection in aelf doctor, a --backfill-ingest option and a --prune-noise garbage-collection pass (#1011, #1029).

A pollution-recovery benchmark measures which facts survive in a store that holds a large volume of keyword-overlapping document chunks. Lexical-match facts survive through BM25. Entity-match facts survive through the L2.5 entity index. Both have recall@5 = 1.0. A fact that shares no term and no entity with the query is not retrieved at all, and the repair for that case is locking.

Two smaller changes also shipped. The release now token-caps the first-turn <core> injection. A 25.9k-belief store dropped from ~733 KB to ~42 KB. The system also counts corroboration per distinct source and not per re-ingest.

v3.8 to v4.0: the lanes were not on the production path

v4.0.0 (2026-07-07) corrects an embarrassing gap. The project set several of the retrieval features above to default-on in retrieve_v2. Those features are the intentional-clustering pack stage, the structural HRR lane and the newer staged lanes. retrieve_v2 is the pipeline that the benchmarks and the evaluation suite call. The production hook path is the code that runs when a live session retrieves memory. That path still called the legacy retrieve(). The features were real. The project gated them and measured them. No user received them.

#1107 closed the gap. retrieve() is now a thin adapter over retrieve_v2. An equivalence test with every staged lane forced off pinned the cutover as byte-identical. The lanes then moved onto the live path one at a time. Each lane moved behind its own latency gate and no-regression gate.

Four lanes reached production:

• Temporal spine (#1064). This lane gives the largest retrieval-coverage gain measured on this codebase. It is deterministic and embeddings-free. The system writes per-session TEMPORAL_NEXT chains at ingest. The lane walks those chains from the top keyword seeds and appends the chronological neighbours. It reaches gold content that shares no salient term with the question, through chronological adjacency. This is the exact hole that the pollution-recovery benchmark (v3.8, above) left open. A fresh LoCoMo sample confirmed the result: +14.6pp gold-set coverage (0.460 → 0.606; temporal questions +17.2pp, multi-hop +10.4pp). That gain is 10× a seeded shuffled control. The out-of-sample gain also exceeded the development gain (+12.7pp on LongMemEval), which is the opposite of winner's curse.
• Junk demotion (#1096). This is a small log-additive rerank penalty. It applies to a belief whose only referential grounding is a transient coordination token, such as a bare PR number or a version tag. It does not apply to a belief grounded in a durable entity, such as a file path, an error code or a symbol. On 118 hand-labeled coordination beliefs, the grounding score separates durable content from ephemeral status at a mean of 0.56 against 0.06. The area under the curve (AUC) for ranking durable content above ephemeral content rose from 0.48 (posterior-only) to 0.87. The penalty is a demotion only. It never changes a well-grounded belief or an entity-free belief.
• Intentional clustering (#436). This is the multi-fact pack stage, and it now reaches users. Cluster coverage moved from 0.500 to 1.000 on the public ablation, and the recall did not degrade. The p99 latency is 0.328 ms against a 5 ms budget.
• Structural HRR (#152). Marker queries (CONTRADICTS:<belief-id>) now go through the graph lane on the production path. The public ablation is constructed so that the answers share no vocabulary with the query. Recall moved from 0.000 to 1.000, because keyword search cannot reach a structural answer by definition.

Two lanes deliberately did not reach production. LoCoMo refuted origin-priority reranking (#1013). The failure there was BM25 recall, because the trusted fact never became a candidate, and reranking cannot correct that. Origin-priority reranking therefore survives only as a within-tier tie-break, default-off. HRR-expand also stays off, because of the +0.13pp ablation in the section above.

The same distrust of recurrence drove the largest scoring fix of the release. Retrieval exposure no longer updates the posterior (#1086). A retrieval had counted as positive evidence, so a frequently retrieved belief continued to rise. On a real 24,883-belief store, junk such as session scaffolding and fragments accumulated ~3× the exposure of clean beliefs. That junk then outscored the clean beliefs, at a mean μ of 0.554 against 0.446. Exposure is now audit-only. The event still enters the feedback history, but the posterior moves only on real signal. The controlled check gives the result: a belief retrieved 50 times no longer ranks above a belief seen once. Under the old behaviour that belief inflated to μ 0.857.

The rest of the release, with the numbers where they exist:

• Curation gained honest tooling. aelf introspect shows the per-session posterior, the recurrence, the grounding and the noise signals. aelf retire and restore make soft-delete reversible end-to-end (#1081).
• The memory files of the host harness now reconcile into the belief graph by default (#1089).
• Dispatched subagents inherit the memory context. Before this change they ran with no access to the store (#1068).
• Valence propagation is now on (#1058). It shipped in v0.1.0, and no production path invoked it until now. The mean fan-out is 1.7 recipients per explicit event, and the walk p99 is < 1 ms.
• Codex joined Claude as a fully supported host (#1052–#1055).

The wide-retrieval knobs gave one more result (#1045). A rise of the BM25 candidate cap from 50 to 200 with an 8,000-token budget takes LongMemEval-S from 58.8 percent to 68.6 percent. That score is above the GPT-4o paper baseline of 60.6 percent. The multi-hop misses were a recall-cap problem and not a ranking problem. The budget alone is inert, because the candidates cap before the pack trim. The test count moved from 4,757 to 6,009.

One research thread closed negative in this cycle. A posterior-informed reranking term (ζ) showed a real directional signal on the development data (permutation p ≈ 5×10⁻⁴, +0.023 MRR). The pre-registered confirmatory run on fresh samples returned −0.001 MRR. The direction was real. The effect size was winner's curse. ζ stays off. The project parked the pull request. The thread reopens only on a new signal.

What is still not solved

Nobody has run a full canonical benchmark suite since v3.0.1 (above). The v4.0 cycle ran targeted fresh-data measurements instead. Those measurements are the temporal-spine coverage gate on a fresh LoCoMo sample, the wide-retrieval knob on LongMemEval-S, and the fully-fed HRR ablation. Suite-wide numbers on the current substrate are still outstanding. A v2 reproducibility harness reproduces 6 of 11 adapters within tolerance.

Known limitations that remain from Section 7:

• Contradiction detection during retrieval. The project has still not wired this detection into the scoring. The system does write a CONTRADICTS semantic-edge substrate at ingest (v3.7, default-off, with incremental per-turn detection). From v4.0, an explicit structural query (CONTRADICTS:<belief-id>) reaches those edges on the production path. Free-text retrieval scoring still does not use those edges for internal consistency.
• Correction rates. Nobody measured the correction rates again. The v1 longitudinal analysis found no decrease. Whether the v3 and v4 changes affect the correction rates is unknown.
• LongMemEval multi-session was 24.1 percent in v1 and 56.4 percent at the v3.0.1 re-run. The wide-retrieval knobs later took LongMemEval-S overall to 68.6 percent. That is an opt-in configuration, and not the default of the hook path. Nobody has measured the score of the default configuration on the current substrate.

The project did solve cross-project scoping. v3 added shared scopes with content-hash deduplication. The 25 percent cross-project noise of the A/B test is no longer expected.

Appendix A: benchmark methodology

This appendix gives the exact protocol for each benchmark. Any deviation invalidates the results. The project developed this protocol after two contamination incidents during development. The public repository contains the full protocol, the contamination verification script and all benchmark adapters.

Contamination protocol

The project identified three contamination modes during development:

1. Ground truth in retrieval output. The retrieval JSON contained answer fields. The LLM reader saw the correct answers while it generated the predictions. This produced the invalid 87.8 percent LoCoMo score. The contamination was not visible immediately, because the first Opus score (61.6 percent F1) was plausible. The project found the contamination only when slow-finishing agents overwrote the merged predictions file. That file produced 87.8 percent F1, near to the human ceiling of 87.9 percent. An exact-match analysis confirmed the contamination: 9 of 10 batches showed 43-87 percent exact-match rates. The project identified four more isolation failures. The first is a renamed _ground_truth field. The second is a set of pre-computed prediction and f1 fields. The third is a set of category_name labels that leak the evaluation strategy. The fourth is no separation between the question context and the scoring metadata. The project retracted all results from this run. Prevention: The adapter code writes two separate files. A mandatory contamination check (verify_clean.py) scans for 30 banned keys before any reader reads the data.

2. LLM self-judging with the answer visible. Prevention: Generation and judging are strictly separate passes.

3. World knowledge override. The LLM reader used real-world knowledge in place of the retrieved context. The LLM reader did this in particular on counterfactual benchmarks. Mitigation: The reader prompts contain explicit instructions to use only the provided context. The project documents this mode as a known limitation (~17 percent of MAB failures).

General protocol

Every benchmark run follows these steps:

Step 1: Data acquisition. Download the dataset from the published source. Verify the row counts and the field names.

Step 2: Retrieval. Run the adapter in --retrieve-only mode. This step produces a retrieval file and a ground truth file. Each test case uses a new SQLite database.

uv run python benchmarks/<adapter>.py \
  --retrieve-only /tmp/benchmark_<name>.json

Step 3: Contamination check. Run this check before any reader reads the data. This step is mandatory.

uv run python benchmarks/verify_clean.py /tmp/benchmark_<name>.json

Step 4: Answer generation. The LLM reader receives only the retrieval file. The reader never sees the ground truth.

Step 5: Scoring. The scoring script reads the predictions and the ground truth. The metrics follow the exact published formulas.

Step 6: Reporting. The report contains these items:

• the exact commands
• the output of the contamination check
• the commit hash of the adapter
• the dataset version
• the reader model
• the scoring metric
• the published baselines
• the known limitations

Per-benchmark specifics

LoCoMo ([Maharana et al., ACL 2024](https://snap-research.github.io/locomo/))

• Dataset: locomo10.json, 10 conversations, 5,882 turns, 1,986 question-and-answer pairs across 5 categories.
• Ingestion: All 10 conversations go through the standard onboarding pipeline. The pipeline preserves the session boundaries.
• Retrieval: FTS5 + HRR + BFS, 2,000-token budget, batch size 1.
• Reader model: Claude Opus 4.6.
• Prompts: The run uses the exact LoCoMo protocol prompts. Categories 1/3/4 use this prompt: "Based on the above context, write an answer in the form of a short phrase..." Category 2 appends this text: "Use DATE of CONVERSATION to answer with an approximate date." Category 5 uses this forced choice: "(a) Not mentioned (b) [adversarial_answer]". The run randomizes the option order (seed=42).
• Scoring: Token-level F1 with Porter stemming and article removal.
• Score: 66.1 percent F1.

MemoryAgentBench FactConsolidation ([Hu et al., ICLR 2026](https://arxiv.org/abs/2507.05257))

• Dataset: HuggingFace ai-hyz/MemoryAgentBench, Conflict_Resolution split.
• Ingestion: The adapter chunks the context at 4,096 tokens. It uses NLTK sent_tokenize and the tiktoken gpt-4o encoding.
• Retrieval (single-hop): FTS5 with triple extraction. The system creates the SUPERSEDES edges automatically.
• Retrieval (multi-hop): The entity-index adapter, with 41 regular-expression patterns. It uses 4-hop chaining with a breadth cap of 30.
• Reader models: Claude Opus 4.6 and Claude Haiku 4.5.
• Scoring: substring_exact_match, as the paper specifies. The multi-hop task also uses chain validation.
• Scores: SH: 90 percent Opus, 62 percent Haiku. MH: 60 percent Opus (raw SEM), 35 percent chain-valid.

StructMemEval ([Shutova et al., 2026](https://github.com/yandex-research/StructMemEval))

• Dataset: GitHub yandex-research/StructMemEval, location/small_bench, 14 cases.
• Ingestion: Narrative timestamps 30 days apart for each session. The run uses the standard pipeline.
• Retrieval: FTS5 with temporal_sort=True.
• Disclosure: The project developed temporal_sort after it saw the initial 29 percent result.
• Score: 14/14 (100 percent).

LongMemEval ([Wu et al., ICLR 2025](https://arxiv.org/abs/2410.10813))

• Dataset: HuggingFace xiaowu0162/longmemeval-cleaned, 500 questions across 6 categories.
• Retrieval: FTS5 + HRR + BFS, 2,000-token budget, top_k=50.
• Judge: The Claude Opus 4.6 binary judge. This is non-standard, because the paper specifies GPT-4o.
• Disclosure: The run uses Opus as the judge in place of GPT-4o. The two results are therefore not directly comparable.
• Score: 59.0 percent (295/500).

Reproducibility

The benchmarks/ directory of the public repository contains all benchmark adapters, all scoring scripts and the contamination verification script. To reproduce any result, run the commands below.

# Clone and install
git clone https://github.com/robotrocketscience/aelfrice
cd aelfrice
uv sync

# Run retrieval (example: MAB single-hop)
uv run python benchmarks/mab_adapter.py \
  --split Conflict_Resolution \
  --source factconsolidation_sh_262k \
  --retrieve-only /tmp/mab_sh.json

# Verify clean
uv run python benchmarks/verify_clean.py /tmp/mab_sh.json

# Score (after running reader)
uv run python benchmarks/exp6_score.py /tmp/mab_sh_preds.json /tmp/mab_sh_gt.json

docs/BENCHMARK_PROTOCOL.md contains the complete per-benchmark commands and the adapter documentation.

Appendix B: literature tables