Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution
  Autonomous Driving VQA from Peru
As multimodal foundational models start being deployed experimentally in
Self-Driving cars, a reasonable question we ask ourselves is how similar to
humans do these systems respond in certain driving situations -- especially
those that are out-of-distribution? To study this, we create the Robusto-1
dataset that uses dashcam video data from Peru, a country with one of the worst
(aggressive) drivers in the world, a high traffic index, and a high ratio of
bizarre to non-bizarre street objects likely never seen in training. In
particular, to preliminarly test at a cognitive level how well Foundational
Visual Language Models (VLMs) compare to Humans in Driving, we move away from
bounding boxes, segmentation maps, occupancy maps or trajectory estimation to
multi-modal Visual Question Answering (VQA) comparing both humans and machines
through a popular method in systems neuroscience known as Representational
Similarity Analysis (RSA). Depending on the type of questions we ask and the
answers these systems give, we will show in what cases do VLMs and Humans
converge or diverge allowing us to probe on their cognitive alignment. We find
that the degree of alignment varies significantly depending on the type of
questions asked to each type of system (Humans vs VLMs), highlighting a gap in
their alignment.
67cff2aa7f2291321730ebfe
Dunant Cusipuma
67cff2aa7f2291321730ebff
David Ortega
67cff2aa7f2291321730ec00
Victor Flores-Benites
67cff2aa7f2291321730ec01
Arturo Deza
claimed_verified
2025-03-12T08:42:25.026Z
61fc521187117e8015dd1173
https://cdn-avatars.huggingface.co/v1/production/uploads/1643926004312-noauth.png
Arturo Deza
user
ArturoDeza
Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution
  Autonomous Driving VQA from Peru
As multimodal foundational models start being deployed experimentally in
Self-Driving cars, a reasonable question we ask ourselves is how similar to
humans do these systems respond in certain driving situations -- especially
those that are out-of-distribution? To study this, we create the Robusto-1
dataset that uses dashcam video data from Peru, a country with one of the worst
(aggressive) drivers in the world, a high traffic index, and a high ratio of
bizarre to non-bizarre street objects likely never seen in training. In
particular, to preliminarly test at a cognitive level how well Foundational
Visual Language Models (VLMs) compare to Humans in Driving, we move away from
bounding boxes, segmentation maps, occupancy maps or trajectory estimation to
multi-modal Visual Question Answering (VQA) comparing both humans and machines
through a popular method in systems neuroscience known as Representational
Similarity Analysis (RSA). Depending on the type of questions we ask and the
answers these systems give, we will show in what cases do VLMs and Humans
converge or diverge allowing us to probe on their cognitive alignment. We find
that the degree of alignment varies significantly depending on the type of
questions asked to each type of system (Humans vs VLMs), highlighting a gap in
their alignment.
Robusto-1 dataset evaluates cognitive alignment between humans and foundational visual language models using multi-modal visual question answering and representational similarity analysis in diverse driving scenarios.
Foundational Visual Language Models VLMs multi-modal Visual Question Answering VQA Representational Similarity Analysis RSA