Multimodal Data and Resource Efficient Device-Directed Speech Detection
  with Large Foundation Models
Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.
6571275f0ea91e592a1cebb5
Dominik Wagner
claimed_verified
2023-12-07T15:10:41.032Z
6311ba6f05cc08a1408d910a
https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
Dominik Wagner
user
dwgnr
6571275f0ea91e592a1cebb6
Alexander Churchill
admin_assigned
2023-12-07T12:23:22.260Z
630fd935dd31b2a8dbe70abf
/avatars/49711a6644a1b8fb13670ac06af87693.svg
Alex Churchill
user
Alexanderchurchill
6571275f0ea91e592a1cebb7
Siddharth Sigtia
6571275f0ea91e592a1cebb8
Panayiotis Georgiou
6571275f0ea91e592a1cebb9
Matt Mirsamadi
6571275f0ea91e592a1cebba
Aarshee Mishra
6571275f0ea91e592a1cebbb
Erik Marchi
Multimodal Data and Resource Efficient Device-Directed Speech Detection
  with Large Foundation Models
Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.
A multimodal system combining ASR and audio encoding improves natural interaction with virtual assistants by reducing the need for trigger phrases and using efficient learning techniques.
automatic speech recognition audio encoder large language model low-rank adaptation prefix tuning equal-error-rate