Structure Beats Scale: What UltraMem Learned on LongMemEval-S
Why better memory systems need time-aware structure, not just larger language models or longer context windows.
Abstract
Long-term memory is becoming one of the central infrastructure problems for AI agents. Most demos make memory look simple: store a fact, retrieve it later, and pass it into context. Real chat assistants face a harder task. They must recall facts stated weeks or months ago, distinguish old information from newer updates, reason across sessions, and answer temporal questions without drowning the model in irrelevant context.
We evaluated UltraMem, an open-source two-layer memory engine, on LongMemEval-S, a benchmark designed to test long-term interactive memory over roughly 50-session chat histories. UltraMem improved from a 50% baseline to 72.5% on a 120-question slice, with particularly strong temporal reasoning at 85%. The largest gains came from memory architecture rather than model scale: round-level chunking, event-time extraction, multi-hop query decomposition, deterministic computation, and a bi-temporal knowledge graph.
The most important finding is that several apparent “reasoning” failures were actually representation failures. When a system has no machine-comparable notion of which fact is latest, prompting the model to “use the latest value” does not solve the problem. Representing facts with event time and supersession semantics does.
These results are indicative rather than leaderboard-official. The evaluation used a 120-question slice and Gemini 2.5 Flash as judge, while some published comparisons use the full benchmark and GPT-4o judging. Still, the work points to a practical conclusion for agent memory: structure beats scale.
1. Why Long-Term Memory Is Hard
Many memory systems work well on the simplest possible case: a user states a fact, the system stores it, and the assistant recalls it later. That is not the real problem.
The real problem looks more like this:
A user mentions a preference in passing.
Several unrelated conversations happen afterward.
The user later contradicts or updates the preference.
Months later, the assistant must answer using the current value, not the stale one.
LongMemEval was created to test this kind of behavior. Each question hides its evidence in a long chat history, often across dozens of sessions. The benchmark separates memory into six abilities:
CategoryWhat It Testssingle-session-userRecall a fact the user stated oncesingle-session-assistantRecall something the assistant saidsingle-session-preferenceApply a preference expressed in passingknowledge-updateReport the latest value of a fact that changedtemporal-reasoningReason over ordering, dates, and elapsed timemulti-sessionAggregate evidence across multiple sessions
That taxonomy matters because “memory” is not one skill. A system can be good at recalling a single user fact and still fail badly at updates, preference application, or temporal reasoning.
2. System Under Test: UltraMem
UltraMem is a self-hostable memory engine written in Rust. It has two main layers.
The document layer chunks content, embeds it, stores it in a vector database, retrieves with hybrid dense and sparse search, and reranks results with a cross-encoder. This layer answers the familiar retrieval question: what pieces of text are probably relevant?
The memory layer distills atomic facts from documents and reconciles those facts over time. It drops duplicates, marks contradictions as updates, flips older values out of the “latest” set, and preserves enrichments as extensions. This layer answers a different question: what does the system believe is true, and what has changed?
The thesis behind UltraMem is that long-term memory cannot be reduced to chunk retrieval. Retrieval can surface evidence, but memory also needs maintained state.
3. Method
The evaluation was designed around one principle: every aggregate score should be explainable at the question level.
UltraMem ingested the benchmark haystacks once, then evaluated multiple answer-logic changes against the same index. This split mattered because ingestion was slow, while scoring changes could be repeated quickly.
Each question was logged with the question, gold answer, model answer, judge verdict, retrieved sessions, and a gold_retrieved flag. That flag separated failures into two classes:
Retrieval miss: the system did not surface the evidence.
Synthesis failure: the system surfaced the evidence, but the answer logic failed.
This distinction changed the direction of the work. After retrieval reached 117 of 120 gold sessions retrieved, most remaining failures were synthesis failures. The system no longer primarily needed more recall. It needed better structure.
4. Results
UltraMem moved from a 50% baseline to 72.5% on a 120-question slice.
The most important category-level result was knowledge-update. Prompt improvements alone left knowledge-update flat at 60%. Adding a bi-temporal graph moved it to 80%.
StageOverallKnowledge UpdateBaseline50.0%60%Prompt improvements66.7%60%Bi-temporal graph72.5%80%
The final category profile was:
CategoryUltraMemsingle-session-user90%single-session-assistant70%single-session-preference45%knowledge-update80%temporal-reasoning85%multi-session65%
Against published field figures, UltraMem appears mid-pack overall, strong on temporal reasoning, competitive on knowledge-update, and weaker on assistant recall and preference. This comparison should be read carefully because the evaluation settings are not identical across systems.
5. What Actually Improved the System
5.1 Round-Level Chunking
The first major fix was almost mundane. Early retrieval failures often happened because conversational content was chunked by paragraph. A user’s answer could land in a chunk that did not match the later question.
UltraMem switched to one user-assistant round per chunk and enriched each embedding key with distilled facts already extracted from the content. This pushed single-session recall toward the ceiling.
The lesson was simple: the model was not failing because it lacked intelligence. The system had not put the right evidence in front of it.
5.2 Event-Time Extraction
Temporal questions care about when something happened, not when it was discussed. UltraMem initially stamped facts with conversation time. That produced absurd date arithmetic.
The fix was to extract event time during distillation. If a user said “last Sunday” in a session dated May 20, the system resolved the event to the appropriate date. After this change, wrong-date errors largely disappeared.
The failure mode then shifted from wrong dates to missing second events, which pointed directly to the next improvement.
5.3 Query Decomposition
Multi-hop questions often name two or more events. A single dense vector query tends to retrieve one of them, not both. UltraMem added query decomposition, splitting compound questions into sub-queries, retrieving evidence independently, and unioning the results.
This produced large directional gains in temporal and multi-session questions. It also reinforced a broader point: for multi-hop memory, one embedding is often the wrong abstraction.
5.4 Deterministic Computation
For counting and date arithmetic, UltraMem moved computation out of the model and into Rust. The model extracts structured data; code performs the arithmetic.
This is not glamorous, but it is one of the cleanest design rules in the whole project. Models can explain arithmetic. They should not be trusted to perform it when the system can compute it deterministically.
5.5 Bi-Temporal Knowledge Graph
The central improvement was a bi-temporal knowledge graph.
Before the graph, knowledge-update failures looked like reasoning failures. The model saw both the old and new values in context and still answered with the stale one. Better prompts did not fix it.
After inspecting the logs, the cause became clear: the data itself had no reliable, machine-comparable representation of “latest.”
UltraMem changed the representation. Each fact became a subject-predicate-object edge with two time axes:
Time AxisMeaningEvent timeWhen the fact was true in the worldIngestion timeWhen the system learned it
The graph also distinguishes singular states from accumulating events. A user’s current location, subscription status, or personal best can supersede older values. Trips, meetings, and attended events accumulate.
Once supersession was computed over event time in Rust, “what is the current value?” became deterministic. The model no longer had to infer ordering from an unstructured bag of evidence.
This moved knowledge-update from 60% to 80%.
6. Negative Results
The failures were as informative as the wins.
6.1 Prompting Plateau
Type-aware prompts improved single-session-preference, but did nothing for knowledge-update. This was the most useful null result in the project. It showed that the bottleneck was not instruction-following. The system lacked the representation required to answer the question.
6.2 An Inert Feature
After the graph improvement, UltraMem added date-windowed counting over graph facts. It fired on 0 of 120 questions.
The reason was structural. The graph represented many facts as entity attributes rather than first-class event nodes. For example, a wedding could be scattered across wedding_venue, wedding_month, and wedding_role edges. There was no countable “attended wedding” node.
That feature did not fail because the model was weak. It failed because the schema did not express the object being counted.
6.3 Measurement Noise
One rerun produced an apparent six-point gain, but the relevant code had not changed. A volatile category moved from 45% to 70% with no corresponding implementation change.
This established a measurement wall. At this scale, run-to-run nondeterminism can be large enough to disguise regressions or invent gains. Past a point, multi-run averaging becomes part of the engineering work, not a luxury.
7. Discussion
The strongest conclusion is that memory quality depends more on representation than on raw model strength.
In one comparison, a mid-tier open model and a frontier model produced scores within noise under the same architecture. Their failure profiles were similar. That does not mean model quality never matters. It means that, for this class of long-memory failure, model scale was not the binding constraint.
The second conclusion is that more context can be harmful. UltraMem tried giving full sessions more broadly, and the score regressed. The model began abstaining on questions it had previously answered correctly. This is consistent with lost-in-the-middle behavior: adding more evidence can reduce the model’s ability to use the right evidence.
A memory engine should not maximize context volume. It should maximize the precision and structure of what reaches the model.
8. Practical Takeaways
Structure beats scale. If the memory representation is wrong, a stronger model may reproduce the same failures more fluently.
Some failures are representational, not reasoning failures. If the system cannot compare dates, distinguish stale from current facts, or count entities, the answer is not a better prompt. The answer is a better schema.
More context is not always better. Retrieval exists to avoid dumping the haystack back into the prompt.
Deterministic operations should be deterministic. Date math, counting, supersession, and latest-value resolution should happen in code wherever possible.
Benchmarks need variance estimates. A single run can be directionally useful, but it should not be treated as proof of a small gain.
9. Limitations
These results should not be read as a formal leaderboard submission. The headline run used a 120-question slice, not the full LongMemEval set. It used Gemini 2.5 Flash as judge, while some field comparisons use GPT-4o. The category numbers are useful for engineering direction, but not a strict head-to-head ranking.
Preference scoring also appears partly judge-limited. Some rejected answers were defensible and on-topic, suggesting that improving that category may require either richer evaluation criteria or multiple judges.
Finally, UltraMem’s current graph still lacks first-class event and entity nodes for several multi-session aggregation tasks. The attribute-edge representation solved latest-value resolution but does not yet express every query the benchmark asks.
10. Next Work
The next architectural step is to promote events and entities to first-class graph nodes and add multi-hop traversal fused with vector retrieval and time filtering.
That should address the remaining weaknesses in multi-session aggregation and countable event queries. Before chasing those improvements, the evaluation itself needs to be pinned with averaged runs. The system is now close enough to the noise floor that measurement quality gates progress.
Conclusion
UltraMem’s LongMemEval-S run suggests a practical direction for agent memory systems: do not treat memory as a bigger prompt. Treat it as a structured, time-aware model of the user’s world.
Retrieval finds evidence. A memory system has to do more. It has to know which facts supersede others, which events accumulate, which dates matter, and which operations should be computed before the model writes a single word.
The lesson is not that models are unimportant. It is that long-term memory fails when the system asks the model to infer structure the data never encoded.
Structure beats scale.
References
Wu et al., “LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory”, ICLR 2025: https://arxiv.org/abs/2410.10813
Rasmussen et al., “Zep: A Temporal Knowledge Graph Architecture for Agent Memory”: https://arxiv.org/abs/2501.13956
Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”, TACL 2024: https://arxiv.org/abs/2307.03172
Yu et al., “Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models”: https://arxiv.org/abs/2311.09210
UltraMem repository and reproducible harness: https://github.com/Akpughe/ultramem