A Touch, Vision, and Language Dataset for Multimodal Alignment
Touch is an important sensing modality for humans, but it has not yet been
incorporated into a multimodal generative language model. This is partially due
to the difficulty of obtaining natural language labels for tactile data and the
complexity of aligning tactile readings with both visual observations and
language descriptions. As a step towards bridging that gap, this work
introduces a new dataset of 44K in-the-wild vision-touch pairs, with English
language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V
(90%). We use this dataset to train a vision-language-aligned tactile encoder
for open-vocabulary classification and a touch-vision-language (TVL) model for
text generation using the trained encoder. Results suggest that by
incorporating touch, the TVL model improves (+29% classification accuracy)
touch-vision-language alignment over existing models trained on any pair of
those modalities. Although only a small fraction of the dataset is
human-labeled, the TVL model demonstrates improved visual-tactile understanding
over GPT-4V (+12%) and open-source vision-language models (+32%) on a new
touch-vision understanding benchmark. Code and data:
https://tactile-vlm.github.io.
65d561795717d720a0c544fc
Letian Fu
extracted_pending
2024-02-21T02:35:38.909Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
65d561795717d720a0c544fd
Gaurav Datta
65d561795717d720a0c544fe
Huang Huang
65d561795717d720a0c544ff
William Chung-Ho Panitch
65d561795717d720a0c54500
Jaimyn Drake
65d561795717d720a0c54501
Joseph Ortiz
65d561795717d720a0c54502
Mustafa Mukadam
65d561795717d720a0c54503
Mike Lambeta
65d561795717d720a0c54504
Roberto Calandra
65d561795717d720a0c54505
Ken Goldberg
A Touch, Vision, and Language Dataset for Multimodal Alignment
Touch is an important sensing modality for humans, but it has not yet been
incorporated into a multimodal generative language model. This is partially due
to the difficulty of obtaining natural language labels for tactile data and the
complexity of aligning tactile readings with both visual observations and
language descriptions. As a step towards bridging that gap, this work
introduces a new dataset of 44K in-the-wild vision-touch pairs, with English
language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V
(90%). We use this dataset to train a vision-language-aligned tactile encoder
for open-vocabulary classification and a touch-vision-language (TVL) model for
text generation using the trained encoder. Results suggest that by
incorporating touch, the TVL model improves (+29% classification accuracy)
touch-vision-language alignment over existing models trained on any pair of
those modalities. Although only a small fraction of the dataset is
human-labeled, the TVL model demonstrates improved visual-tactile understanding
over GPT-4V (+12%) and open-source vision-language models (+32%) on a new
touch-vision understanding benchmark. Code and data:
https://tactile-vlm.github.io.
The study presents a new dataset and multimodal model that integrates touch, vision, and language, showing improvements in touch-vision-language alignment and visual-tactile understanding.
multimodal generative language model tactile data vision-touch pairs English language labels GPT-4V textual pseudo-labels vision-language-aligned tactile encoder touch-vision-language model text generation open-vocabulary classification touch-vision understanding benchmark