Embodied Cognition Augmented End2End Autonomous Driving
Ling Niu, Xiaoji Zheng, han wang, Ziyuan Yang, Chen Zheng, Bokui Chen, Jiangtao Gong
In recent years, vision-based end-to-end autonomous driving has emerged as a new paradigm. However, mainstream end-to-end approaches typically rely on visual feature extraction networks trained under label supervision, and this limited supervision scheme restricts the generality and applicability of such models. In this paper, we propose a novel paradigm, termed $E^{3}AD$, which advocates for cross-modal self-supervised learning between visual feature extraction networks and a general EEG large model, in order to learn latent human driving cognition for enhancing end-to-end planning. In this work, self-collected cognitive data are utilized in the aforementioned cross-modal self-supervised learning framework. Subsequently, we investigate the approaches and potential mechanisms of augmenting end-to-end planning with human driving cognition on mainstream end-to-end autonomous driving benchmark models, leveraging well-established autonomous driving datasets. Both open-loop tests and closed-loop simulations are conducted for a comprehensive evaluation of planning performance. Experimental results demonstrate that the $E^{3}AD$ paradigm significantly enhances the end-to-end planning performance of baseline models. Ablation studies further validate the contribution of driving cognition and the effectiveness of self-supervised training. To the best of our knowledge, this is the first work to integrate human driving cognition for improving end-to-end autonomous driving planning. It represents an initial attempt to incorporate embodied cognitive data into end-to-end autonomous driving, providing valuable insights for future brain-inspired autonomous driving systems