Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
  Learning Language-Agnostic Speech Representations
We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
framework, dubbed Zero-AVSR, which enables speech recognition in target
languages without requiring any audio-visual speech data in those languages.
Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
which learns language-agnostic speech representations by predicting Roman text.
Then, by leveraging the strong multilingual modeling capabilities of Large
Language Models (LLMs), we propose converting the predicted Roman text into
language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
a step further, we explore a unified Zero-AVSR approach by directly integrating
the audio-visual speech representations encoded by the AV-Romanizer into the
LLM. This is achieved through finetuning the adapter and the LLM using our
proposed multi-task learning scheme. To capture the wide spectrum of phonetic
and linguistic diversity, we also introduce a Multilingual Audio-Visual
Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
across 82 languages, along with transcriptions in both language-specific
graphemes and Roman text. Extensive analysis and experiments confirm that the
proposed Zero-AVSR framework has the potential to expand language support
beyond the languages seen during the training of the AV-Romanizer.
67cfa7ca62df3125810efc27
Jeong Hun Yeo
claimed_verified
2025-03-11T10:57:57.858Z
65bb389bc60df511e142ba82
/avatars/5884ea11d2682ed71585a4025a4248c1.svg
Jeong Hun Yeo
user
JeongHun0716
67cfa7ca62df3125810efc28
Minsu Kim
67cfa7ca62df3125810efc29
Chae Won Kim
67cfa7ca62df3125810efc2a
Stavros Petridis
67cfa7ca62df3125810efc2b
Yong Man Ro
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
  Learning Language-Agnostic Speech Representations
We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
framework, dubbed Zero-AVSR, which enables speech recognition in target
languages without requiring any audio-visual speech data in those languages.
Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
which learns language-agnostic speech representations by predicting Roman text.
Then, by leveraging the strong multilingual modeling capabilities of Large
Language Models (LLMs), we propose converting the predicted Roman text into
language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
a step further, we explore a unified Zero-AVSR approach by directly integrating
the audio-visual speech representations encoded by the AV-Romanizer into the
LLM. This is achieved through finetuning the adapter and the LLM using our
proposed multi-task learning scheme. To capture the wide spectrum of phonetic
and linguistic diversity, we also introduce a Multilingual Audio-Visual
Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
across 82 languages, along with transcriptions in both language-specific
graphemes and Roman text. Extensive analysis and experiments confirm that the
proposed Zero-AVSR framework has the potential to expand language support
beyond the languages seen during the training of the AV-Romanizer.
A novel zero-shot Audio-Visual Speech Recognition (AVSR) framework, Zero-AVSR, uses audio-visual speech representations and large language models to predict speech in unseen languages, facilitated by a multilingual dataset.
Audio-Visual Speech Recognition AVSR Audio-Visual Speech Romanizer AV-Romanizer language-agnostic speech representations Large Language Models LLMs Cascaded Zero-AVSR unified Zero-AVSR adapter multi-task learning Multilingual Audio-Visual Romanized Corpus MARC multilingual dataset