MEMTRACK: Evaluating Long-Term Memory and State Tracking in
  Multi-Platform Dynamic Agent Environments
Recent works on context and memory benchmarking have primarily focused on
conversational instances but the need for evaluating memory in dynamic
enterprise environments is crucial for its effective application. We introduce
MEMTRACK, a benchmark designed to evaluate long-term memory and state tracking
in multi-platform agent environments. MEMTRACK models realistic organizational
workflows by integrating asynchronous events across multiple communication and
productivity platforms such as Slack, Linear and Git. Each benchmark instance
provides a chronologically platform-interleaved timeline, with noisy,
conflicting, cross-referring information as well as potential
codebase/file-system comprehension and exploration. Consequently, our benchmark
tests memory capabilities such as acquistion, selection and conflict
resolution. We curate the MEMTRACK dataset through both manual expert driven
design and scalable agent based synthesis, generating ecologically valid
scenarios grounded in real world software development processes. We introduce
pertinent metrics for Correctness, Efficiency, and Redundancy that capture the
effectiveness of memory mechanisms beyond simple QA performance. Experiments
across SoTA LLMs and memory backends reveal challenges in utilizing memory
across long horizons, handling cross-platform dependencies, and resolving
contradictions. Notably, the best performing GPT-5 model only achieves a 60\%
Correctness score on MEMTRACK. This work provides an extensible framework for
advancing evaluation research for memory-augmented agents, beyond existing
focus on conversational setups, and sets the stage for multi-agent,
multi-platform memory benchmarking in complex organizational settings
68e6f1237ae125f9582e6835
Darshan Deshpande
claimed_verified
2025-10-09T06:50:50.374Z
60390e04770949ef34f12d9b
https://cdn-avatars.huggingface.co/v1/production/uploads/1632225571380-60390e04770949ef34f12d9b.jpeg
Darshan Deshpande
user
DarshanDeshpande
68e6f1237ae125f9582e6836
Varun Gangal
68e6f1237ae125f9582e6837
Hersh Mehta
68e6f1237ae125f9582e6838
Anand Kannappan
68e6f1237ae125f9582e6839
Rebecca Qian
68e6f1237ae125f9582e683a
Peng Wang
MEMTRACK: Evaluating Long-Term Memory and State Tracking in
  Multi-Platform Dynamic Agent Environments
Recent works on context and memory benchmarking have primarily focused on
conversational instances but the need for evaluating memory in dynamic
enterprise environments is crucial for its effective application. We introduce
MEMTRACK, a benchmark designed to evaluate long-term memory and state tracking
in multi-platform agent environments. MEMTRACK models realistic organizational
workflows by integrating asynchronous events across multiple communication and
productivity platforms such as Slack, Linear and Git. Each benchmark instance
provides a chronologically platform-interleaved timeline, with noisy,
conflicting, cross-referring information as well as potential
codebase/file-system comprehension and exploration. Consequently, our benchmark
tests memory capabilities such as acquistion, selection and conflict
resolution. We curate the MEMTRACK dataset through both manual expert driven
design and scalable agent based synthesis, generating ecologically valid
scenarios grounded in real world software development processes. We introduce
pertinent metrics for Correctness, Efficiency, and Redundancy that capture the
effectiveness of memory mechanisms beyond simple QA performance. Experiments
across SoTA LLMs and memory backends reveal challenges in utilizing memory
across long horizons, handling cross-platform dependencies, and resolving
contradictions. Notably, the best performing GPT-5 model only achieves a 60\%
Correctness score on MEMTRACK. This work provides an extensible framework for
advancing evaluation research for memory-augmented agents, beyond existing
focus on conversational setups, and sets the stage for multi-agent,
multi-platform memory benchmarking in complex organizational settings
MEMTRACK is a benchmark for evaluating long-term memory and state tracking in multi-platform agent environments, focusing on dynamic enterprise settings and providing metrics for correctness, efficiency, and redundancy.
MEMTRACK long-term memory state tracking multi-platform agent environments asynchronous events Slack Linear Git chronological timeline noisy information conflicting information cross-referring information codebase comprehension file-system exploration memory acquisition memory selection conflict resolution MEMTRACK dataset Correctness Efficiency Redundancy SoTA LLMs memory backends GPT-5 multi-agent multi-platform memory benchmarking