Masked Transformers with Test-Time Adaptation for Speech Neuroprostheses
Ebrahim Feghhi, Shreyas Kaasyap, Nima Hadidi, Jonathan Kao
Speech neuroprostheses offer a path to restore communication in patients with severe paralysis by decoding speech directly from neural activity. To function seamlessly, speech neuroprostheses should ideally be (1) accurate, (2) capable of real-time decoding, (3) efficient in terms of memory and computation, and (4) robust to distribution shifts in neural actvity over time. Prior work on the Brain-to-Text Benchmark improved the accuracy of a baseline gated recurrent unit (GRU) decoder by integrating the outputs of an ensemble of GRUs with large language models. However, these gains come at the expense of criteria (2) and (3). In this work, we make holistic improvements to the baseline through two core contributions: masking contiguous chunks of neural activity during training, and replacing the GRU with a lightweight Transformer. Together, these improvements yield a $20$\% reduction in word error rate while substantially reducing memory requirements and per-epoch training times. Finally, leveraging the Transformer’s suitability for test-time adaptation, we show improved robustness to across-day distribution shifts by calibrating the decoder to make confident and consistent predictions across time-masked versions of the same input during inference.