OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks
Large language models excel at abstract reasoning but their capacity for
embodied agent reasoning remains largely unexplored. We present OmniEAR, a
comprehensive framework for evaluating how language models reason about
physical interactions, tool usage, and multi-agent coordination in embodied
tasks. Unlike existing benchmarks that provide predefined tool sets or explicit
collaboration directives, OmniEAR requires agents to dynamically acquire
capabilities and autonomously determine coordination strategies based on task
demands. Through text-based environment representation, we model continuous
physical properties and complex spatial relationships across 1,500 scenarios
spanning household and industrial domains. Our systematic evaluation reveals
severe performance degradation when models must reason from constraints: while
achieving 85-96% success with explicit instructions, performance drops to
56-85% for tool reasoning and 63-85% for implicit collaboration, with compound
tasks showing over 50% failure rates. Surprisingly, complete environmental
information degrades coordination performance, indicating models cannot filter
task-relevant constraints. Fine-tuning improves single-agent tasks dramatically
(0.6% to 76.3%) but yields minimal multi-agent gains (1.5% to 5.5%), exposing
fundamental architectural limitations. These findings demonstrate that embodied
reasoning poses fundamentally different challenges than current models can
address, establishing OmniEAR as a rigorous benchmark for evaluating and
advancing embodied AI systems. Our code and data are included in the
supplementary materials and will be open-sourced upon acceptance.
6895644d48b0ae5ca2710d64
Zixuan Wang
claimed_verified
2025-08-08T16:19:35.707Z
677d3523a6918748bf81d0e9
/avatars/c9961ac54d089efb36db20c421c2bea2.svg
wangzixuan
user
wangzx1210
6895644d48b0ae5ca2710d65
Dingming Li
6895644d48b0ae5ca2710d66
Hongxing Li
6895644d48b0ae5ca2710d67
Shuo Chen
6895644d48b0ae5ca2710d68
Yuchen Yan
claimed_verified
2025-08-11T06:50:03.727Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
6895644d48b0ae5ca2710d69
Wenqi Zhang
6895644d48b0ae5ca2710d6a
Yongliang Shen
claimed_verified
2025-08-11T06:50:07.402Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
6895644d48b0ae5ca2710d6b
Weiming Lu
6895644d48b0ae5ca2710d6c
Jun Xiao
6895644d48b0ae5ca2710d6d
Yueting Zhuang
OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks
Large language models excel at abstract reasoning but their capacity for
embodied agent reasoning remains largely unexplored. We present OmniEAR, a
comprehensive framework for evaluating how language models reason about
physical interactions, tool usage, and multi-agent coordination in embodied
tasks. Unlike existing benchmarks that provide predefined tool sets or explicit
collaboration directives, OmniEAR requires agents to dynamically acquire
capabilities and autonomously determine coordination strategies based on task
demands. Through text-based environment representation, we model continuous
physical properties and complex spatial relationships across 1,500 scenarios
spanning household and industrial domains. Our systematic evaluation reveals
severe performance degradation when models must reason from constraints: while
achieving 85-96% success with explicit instructions, performance drops to
56-85% for tool reasoning and 63-85% for implicit collaboration, with compound
tasks showing over 50% failure rates. Surprisingly, complete environmental
information degrades coordination performance, indicating models cannot filter
task-relevant constraints. Fine-tuning improves single-agent tasks dramatically
(0.6% to 76.3%) but yields minimal multi-agent gains (1.5% to 5.5%), exposing
fundamental architectural limitations. These findings demonstrate that embodied
reasoning poses fundamentally different challenges than current models can
address, establishing OmniEAR as a rigorous benchmark for evaluating and
advancing embodied AI systems. Our code and data are included in the
supplementary materials and will be open-sourced upon acceptance.
OmniEAR evaluates language models' embodied reasoning capabilities in physical interactions, tool usage, and multi-agent coordination, revealing performance degradation under constraints and highlighting architectural limitations.
embodied agent reasoning OmniEAR text-based environment representation continuous physical properties complex spatial relationships tool reasoning implicit collaboration fine-tuning embodied AI systems