LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive
  Memory
Recent large language model (LLM)-driven chat assistant systems have
integrated memory components to track user-assistant chat histories, enabling
more accurate and personalized responses. However, their long-term memory
capabilities in sustained interactions remain underexplored. This paper
introduces LongMemEval, a comprehensive benchmark designed to evaluate five
core long-term memory abilities of chat assistants: information extraction,
multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
With 500 meticulously curated questions embedded within freely scalable
user-assistant chat histories, LongMemEval presents a significant challenge to
existing long-term memory systems, with commercial chat assistants and
long-context LLMs showing 30% accuracy drop on memorizing information across
sustained interactions. We then present a unified framework that breaks down
the long-term memory design into four design choices across the indexing,
retrieval, and reading stages. Built upon key experimental insights, we propose
several memory designs including session decomposition for optimizing value
granularity, fact-augmented key expansion for enhancing the index structure,
and time-aware query expansion for refining the search scope. Experiment
results show that these optimizations greatly improve both memory recall and
downstream question answering on LongMemEval. Overall, our study provides
valuable resources and guidance for advancing the long-term memory capabilities
of LLM-based chat assistants, paving the way toward more personalized and
reliable conversational AI.
670defb480363ae01368542c
Di Wu
claimed_verified
2024-10-15T08:07:50.570Z
639bf367445b133a4e97ef9c
/avatars/51b59f4616a01796e07c05c9aa5286f8.svg
Di Wu
user
xiaowu0162
670defb480363ae01368542d
Hongwei Wang
670defb480363ae01368542e
Wenhao Yu
admin_assigned
2024-10-15T16:41:35.824Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
670defb480363ae01368542f
Yuwei Zhang
claimed_verified
2024-10-15T21:17:57.174Z
663187c1a2354b0f50ab10a0
/avatars/bef0fd9d2afa6a4990bc32bd55cbe163.svg
Yuwei Zhang
user
YWZBrandon
670defb480363ae013685430
Kai-Wei Chang
admin_assigned
2024-10-15T16:42:29.388Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
670defb480363ae013685431
Dong Yu
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive
  Memory
Recent large language model (LLM)-driven chat assistant systems have
integrated memory components to track user-assistant chat histories, enabling
more accurate and personalized responses. However, their long-term memory
capabilities in sustained interactions remain underexplored. This paper
introduces LongMemEval, a comprehensive benchmark designed to evaluate five
core long-term memory abilities of chat assistants: information extraction,
multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
With 500 meticulously curated questions embedded within freely scalable
user-assistant chat histories, LongMemEval presents a significant challenge to
existing long-term memory systems, with commercial chat assistants and
long-context LLMs showing 30% accuracy drop on memorizing information across
sustained interactions. We then present a unified framework that breaks down
the long-term memory design into four design choices across the indexing,
retrieval, and reading stages. Built upon key experimental insights, we propose
several memory designs including session decomposition for optimizing value
granularity, fact-augmented key expansion for enhancing the index structure,
and time-aware query expansion for refining the search scope. Experiment
results show that these optimizations greatly improve both memory recall and
downstream question answering on LongMemEval. Overall, our study provides
valuable resources and guidance for advancing the long-term memory capabilities
of LLM-based chat assistants, paving the way toward more personalized and
reliable conversational AI.
LongMemEval assesses long-term memory in chat assistants through five core abilities, identifying gaps and proposing memory design optimizations that enhance recall and question answering.
long-term memory LongMemEval information extraction multi-session reasoning temporal reasoning knowledge updates abstention indexing retrieval reading stages session decomposition fact-augmented key expansion time-aware query expansion