Towards Embodied Cognition in Robots via Spatially Grounded Synthetic
  Worlds
We present a conceptual framework for training Vision-Language Models (VLMs)
to perform Visual Perspective Taking (VPT), a core capability for embodied
cognition essential for Human-Robot Interaction (HRI). As a first step toward
this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse,
that enables supervised learning for spatial reasoning tasks. Each instance
includes an RGB image, a natural language description, and a ground-truth 4X4
transformation matrix representing object pose. We focus on inferring Z-axis
distance as a foundational skill, with future extensions targeting full 6
Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to
support further research. This work serves as a foundational step toward
embodied AI systems capable of spatial understanding in interactive human-robot
scenarios.
682dd1aaf3e4b74b546b6db4
Joel Currie
extracted_confirmed
2025-05-21T13:15:52.516Z
6793cc6e09bb7d2e0441a335
https://cdn-avatars.huggingface.co/v1/production/uploads/6793cc6e09bb7d2e0441a335/QtAEDNeEJFvJuqbvvkvj3.jpeg
Joel Currie
user
jwgcurrie
682dd1aaf3e4b74b546b6db5
Gioele Migno
682dd1aaf3e4b74b546b6db6
Enrico Piacenti
extracted_pending
2025-05-21T13:14:19.458Z
67d9304651046c588116e552
/avatars/796426f7a7e024fba594b0c56ece8a57.svg
Enrico Piacenti
user
ep10102
682dd1aaf3e4b74b546b6db7
Maria Elena Giannaccini
682dd1aaf3e4b74b546b6db8
Patric Bach
682dd1aaf3e4b74b546b6db9
Davide De Tommaso
682dd1aaf3e4b74b546b6dba
Agnieszka Wykowska
Towards Embodied Cognition in Robots via Spatially Grounded Synthetic
  Worlds
We present a conceptual framework for training Vision-Language Models (VLMs)
to perform Visual Perspective Taking (VPT), a core capability for embodied
cognition essential for Human-Robot Interaction (HRI). As a first step toward
this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse,
that enables supervised learning for spatial reasoning tasks. Each instance
includes an RGB image, a natural language description, and a ground-truth 4X4
transformation matrix representing object pose. We focus on inferring Z-axis
distance as a foundational skill, with future extensions targeting full 6
Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to
support further research. This work serves as a foundational step toward
embodied AI systems capable of spatial understanding in interactive human-robot
scenarios.
A synthetic dataset in NVIDIA Omniverse aids in training Vision-Language Models for Visual Perspective Taking by providing supervised learning for spatial reasoning tasks.
Vision-Language Models Visual Perspective Taking embodied cognition Human-Robot Interaction NVIDIA Omniverse spatial reasoning RGB image natural language description 4X4 transformation matrix object pose Z-axis distance Degrees Of Freedom