Multimodal Data and Resource Efficient Device-Directed Speech Detection
  with Large Foundation Models
Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.
6571275f0ea91e592a1cebb5
Dominik Wagner
claimed_verified
2023-12-07T15:10:41.032Z
6311ba6f05cc08a1408d910a
https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
Dominik Wagner
user
dwgnr
6571275f0ea91e592a1cebb6
Alexander Churchill
admin_assigned
2023-12-07T12:23:22.260Z
630fd935dd31b2a8dbe70abf
/avatars/49711a6644a1b8fb13670ac06af87693.svg
Alex Churchill
user
Alexanderchurchill
6571275f0ea91e592a1cebb7
Siddharth Sigtia
6571275f0ea91e592a1cebb8
Panayiotis Georgiou
6571275f0ea91e592a1cebb9
Matt Mirsamadi
6571275f0ea91e592a1cebba
Aarshee Mishra
6571275f0ea91e592a1cebbb
Erik Marchi
Multimodal Data and Resource Efficient Device-Directed Speech Detection
  with Large Foundation Models
Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.
automatic speech recognition (ASR) decoder signals audio encoder large language model (LLM) low-rank adaptation prefix tuning equal-error-rates (EERs) low-dimensional specialized audio representations high-dimensional general audio representations