Embodied Agents Meet Personalization: Exploring Memory Utilization for
  Personalized Assistance
Embodied agents empowered by large language models (LLMs) have shown strong
performance in household object rearrangement tasks. However, these tasks
primarily focus on single-turn interactions with simplified instructions, which
do not truly reflect the challenges of providing meaningful assistance to
users. To provide personalized assistance, embodied agents must understand the
unique semantics that users assign to the physical world (e.g., favorite cup,
breakfast routine) by leveraging prior interaction history to interpret
dynamic, real-world instructions. Yet, the effectiveness of embodied agents in
utilizing memory for personalized assistance remains largely underexplored. To
address this gap, we present MEMENTO, a personalized embodied agent evaluation
framework designed to comprehensively assess memory utilization capabilities to
provide personalized assistance. Our framework consists of a two-stage memory
evaluation process design that enables quantifying the impact of memory
utilization on task performance. This process enables the evaluation of agents'
understanding of personalized knowledge in object rearrangement tasks by
focusing on its role in goal interpretation: (1) the ability to identify target
objects based on personal meaning (object semantics), and (2) the ability to
infer object-location configurations from consistent user patterns, such as
routines (user patterns). Our experiments across various LLMs reveal
significant limitations in memory utilization, with even frontier models like
GPT-4o experiencing a 30.5% performance drop when required to reference
multiple memories, particularly in tasks involving user patterns. These
findings, along with our detailed analyses and case studies, provide valuable
insights for future research in developing more effective personalized embodied
agents. Project website: https://connoriginal.github.io/MEMENTO
6835365d2925bc8bb23a57c7
Taeyoon Kwon
claimed_verified
2025-05-27T07:50:26.210Z
636b529ef796304dd67d139c
/avatars/7a64d5095fcb1da558b52ad48177ad76.svg
Taeyoon Kwon
user
Connoriginal
6835365d2925bc8bb23a57c8
Dongwook Choi
claimed_verified
2025-06-16T13:54:37.749Z
654c263fbe11400417c93d9f
/avatars/eb5778e28091200efee2c6b68589a1a2.svg
choi dongwook
user
dongwookchoi
6835365d2925bc8bb23a57c9
Sunghwan Kim
6835365d2925bc8bb23a57ca
Hyojun Kim
6835365d2925bc8bb23a57cb
Seungjun Moon
admin_assigned
2025-05-27T08:24:40.306Z
6420f4f55bccaa42484496e5
/avatars/4996ba26955f8423c946b1ecd3989964.svg
Seung Jun Moon
user
Lune-Blue
6835365d2925bc8bb23a57cc
Beong-woo Kwak
admin_assigned
2025-05-27T08:24:46.911Z
64b72a408ba7d6c922c73054
/avatars/6d9797430bc36f05fb950b84aa6a9374.svg
Beong Woo Kwak
user
bwookwak
6835365d2925bc8bb23a57cd
Kuan-Hao Huang
admin_assigned
2025-05-27T08:24:53.502Z
658a57b4126b8d7eae07b983
https://cdn-avatars.huggingface.co/v1/production/uploads/658a57b4126b8d7eae07b983/reZrbaObwq_kNVsuTR_iR.jpeg
Kuan-Hao Huang
user
kuanhaoh
6835365d2925bc8bb23a57ce
Jinyoung Yeo
admin_assigned
2025-05-27T08:25:01.610Z
682e91865fa8c5df85b3d8e5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XWTfZoOjCdMnqaFEBBYWe.png
Jinyoung Yeo
user
jinyeo
Embodied Agents Meet Personalization: Exploring Memory Utilization for
  Personalized Assistance
Embodied agents empowered by large language models (LLMs) have shown strong
performance in household object rearrangement tasks. However, these tasks
primarily focus on single-turn interactions with simplified instructions, which
do not truly reflect the challenges of providing meaningful assistance to
users. To provide personalized assistance, embodied agents must understand the
unique semantics that users assign to the physical world (e.g., favorite cup,
breakfast routine) by leveraging prior interaction history to interpret
dynamic, real-world instructions. Yet, the effectiveness of embodied agents in
utilizing memory for personalized assistance remains largely underexplored. To
address this gap, we present MEMENTO, a personalized embodied agent evaluation
framework designed to comprehensively assess memory utilization capabilities to
provide personalized assistance. Our framework consists of a two-stage memory
evaluation process design that enables quantifying the impact of memory
utilization on task performance. This process enables the evaluation of agents'
understanding of personalized knowledge in object rearrangement tasks by
focusing on its role in goal interpretation: (1) the ability to identify target
objects based on personal meaning (object semantics), and (2) the ability to
infer object-location configurations from consistent user patterns, such as
routines (user patterns). Our experiments across various LLMs reveal
significant limitations in memory utilization, with even frontier models like
GPT-4o experiencing a 30.5% performance drop when required to reference
multiple memories, particularly in tasks involving user patterns. These
findings, along with our detailed analyses and case studies, provide valuable
insights for future research in developing more effective personalized embodied
agents. Project website: https://connoriginal.github.io/MEMENTO
MEMENTO evaluates personalized memory utilization in embodied agents, revealing limitations in understanding user semantics and routines.
embodied agents large language models (LLMs) object rearrangement tasks user semantics prior interaction history memory utilization personalized assistance goal interpretation object semantics user patterns