Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with
  Long-Term Memory
We introduce M3-Agent, a novel multimodal agent framework equipped with
long-term memory. Like humans, M3-Agent can process real-time visual and
auditory inputs to build and update its long-term memory. Beyond episodic
memory, it also develops semantic memory, enabling it to accumulate world
knowledge over time. Its memory is organized in an entity-centric, multimodal
format, allowing deeper and more consistent understanding of the environment.
Given an instruction, M3-Agent autonomously performs multi-turn, iterative
reasoning and retrieves relevant information from memory to accomplish the
task. To evaluate memory effectiveness and memory-based reasoning in multimodal
agents, we develop M3-Bench, a new long-video question answering benchmark.
M3-Bench comprises 100 newly recorded real-world videos captured from a robot's
perspective (M3-Bench-robot) and 929 web-sourced videos across diverse
scenarios (M3-Bench-web). We annotate question-answer pairs designed to test
key capabilities essential for agent applications, such as human understanding,
general knowledge extraction, and cross-modal reasoning. Experimental results
show that M3-Agent, trained via reinforcement learning, outperforms the
strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o,
achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web
and VideoMME-long, respectively. Our work advances the multimodal agents toward
more human-like long-term memory and provides insights into their practical
design. Model, code and data are available at
https://github.com/bytedance-seed/m3-agent
689d8af7b083e610d741eaaf
Lin Long
689d8af7b083e610d741eab0
Yichen He
689d8af7b083e610d741eab1
Wentao Ye
689d8af7b083e610d741eab2
Yiyuan Pan
689d8af7b083e610d741eab3
Yuan Lin
689d8af7b083e610d741eab4
Hang Li
689d8af7b083e610d741eab5
Junbo Zhao
689d8af7b083e610d741eab6
Wei Li
Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with
  Long-Term Memory
We introduce M3-Agent, a novel multimodal agent framework equipped with
long-term memory. Like humans, M3-Agent can process real-time visual and
auditory inputs to build and update its long-term memory. Beyond episodic
memory, it also develops semantic memory, enabling it to accumulate world
knowledge over time. Its memory is organized in an entity-centric, multimodal
format, allowing deeper and more consistent understanding of the environment.
Given an instruction, M3-Agent autonomously performs multi-turn, iterative
reasoning and retrieves relevant information from memory to accomplish the
task. To evaluate memory effectiveness and memory-based reasoning in multimodal
agents, we develop M3-Bench, a new long-video question answering benchmark.
M3-Bench comprises 100 newly recorded real-world videos captured from a robot's
perspective (M3-Bench-robot) and 929 web-sourced videos across diverse
scenarios (M3-Bench-web). We annotate question-answer pairs designed to test
key capabilities essential for agent applications, such as human understanding,
general knowledge extraction, and cross-modal reasoning. Experimental results
show that M3-Agent, trained via reinforcement learning, outperforms the
strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o,
achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web
and VideoMME-long, respectively. Our work advances the multimodal agents toward
more human-like long-term memory and provides insights into their practical
design. Model, code and data are available at
https://github.com/bytedance-seed/m3-agent
M3-Agent, a multimodal agent with long-term memory, performs multi-turn reasoning and outperforms baselines on a new long-video question answering benchmark.
multimodal agent long-term memory episodic memory semantic memory real-time visual inputs real-time auditory inputs multi-turn reasoning iterative reasoning M3-Bench long-video question answering benchmark reinforcement learning Gemini-1.5-pro GPT-4o human understanding general knowledge extraction cross-modal reasoning