Shortcut Learning in Generalist Robot Policies: The Role of Dataset
  Diversity and Fragmentation
Generalist robot policies trained on large-scale datasets such as Open
X-Embodiment (OXE) demonstrate strong performance across a wide range of tasks.
However, they often struggle to generalize beyond the distribution of their
training data. In this paper, we investigate the underlying cause of this
limited generalization capability. We identify shortcut learning -- the
reliance on task-irrelevant features -- as a key impediment to generalization.
Through comprehensive theoretical and empirical analysis, we uncover two
primary contributors to shortcut learning: (1) limited diversity within
individual sub-datasets, and (2) significant distributional disparities across
sub-datasets, leading to dataset fragmentation. These issues arise from the
inherent structure of large-scale datasets like OXE, which are typically
composed of multiple sub-datasets collected independently across varied
environments and embodiments. Our findings provide critical insights into
dataset collection strategies that can reduce shortcut learning and enhance the
generalization ability of generalist robot policies. Moreover, in scenarios
where acquiring new large-scale data is impractical, we demonstrate that
carefully selected robotic data augmentation strategies can effectively reduce
shortcut learning in existing offline datasets, thereby improving
generalization capabilities of generalist robot policies, e.g., pi_0, in
both simulation and real-world environments. More information at
https://lucky-light-sun.github.io/proj/shortcut-learning-in-grps/.
689ae1cffab6fdd2e52ac58a
Youguang Xing
689ae1cffab6fdd2e52ac58b
Xu Luo
claimed_verified
2025-08-13T07:18:43.805Z
64d19b5431c655ff8aa35081
/avatars/5acdbe28ff841b0df32c3948799ac8f2.svg
Xu Luo
user
FrankLuox
689ae1cffab6fdd2e52ac58c
Junlin Xie
689ae1cffab6fdd2e52ac58d
Lianli Gao
689ae1cffab6fdd2e52ac58e
Hengtao Shen
689ae1cffab6fdd2e52ac58f
Jingkuan Song
Shortcut Learning in Generalist Robot Policies: The Role of Dataset
  Diversity and Fragmentation
Generalist robot policies trained on large-scale datasets such as Open
X-Embodiment (OXE) demonstrate strong performance across a wide range of tasks.
However, they often struggle to generalize beyond the distribution of their
training data. In this paper, we investigate the underlying cause of this
limited generalization capability. We identify shortcut learning -- the
reliance on task-irrelevant features -- as a key impediment to generalization.
Through comprehensive theoretical and empirical analysis, we uncover two
primary contributors to shortcut learning: (1) limited diversity within
individual sub-datasets, and (2) significant distributional disparities across
sub-datasets, leading to dataset fragmentation. These issues arise from the
inherent structure of large-scale datasets like OXE, which are typically
composed of multiple sub-datasets collected independently across varied
environments and embodiments. Our findings provide critical insights into
dataset collection strategies that can reduce shortcut learning and enhance the
generalization ability of generalist robot policies. Moreover, in scenarios
where acquiring new large-scale data is impractical, we demonstrate that
carefully selected robotic data augmentation strategies can effectively reduce
shortcut learning in existing offline datasets, thereby improving
generalization capabilities of generalist robot policies, e.g., pi_0, in
both simulation and real-world environments. More information at
https://lucky-light-sun.github.io/proj/shortcut-learning-in-grps/.
Shortcut learning in generalist robot policies trained on large-scale datasets limits generalization, and this can be mitigated through improved dataset collection and data augmentation strategies.
shortcut learning generalist robot policies Open X-Embodiment dataset fragmentation robotic data augmentation