Interface Design for Self-Supervised Speech Models
Self-supervised speech (SSL) models have recently become widely adopted for
many downstream speech processing tasks. The general usage pattern is to employ
SSL models as feature extractors, and then train a downstream prediction head
to solve a specific task. However, different layers of SSL models have been
shown to capture different types of information, and the methods of combining
them are not well studied. To this end, we extend the general framework for SSL
model utilization by proposing the interface that connects the upstream and
downstream. Under this view, the dominant technique of combining features via a
layerwise weighted sum can be regarded as a specific interface. We propose
several alternative interface designs and demonstrate that the weighted sum
interface is suboptimal for many tasks. In particular, we show that a
convolutional interface whose depth scales logarithmically with the depth of
the upstream model consistently outperforms many other interface designs.
66742e344dd90a98c21db5ba
Yi-Jen Shih
extracted_pending
2024-06-20T13:27:16.850Z
62de905d35a8bfe88fc091a5
/avatars/b9d289319f44231b6e50de491f0b3c30.svg
Ian Shih
user
atosystem
66742e344dd90a98c21db5bb
David Harwath
Interface Design for Self-Supervised Speech Models
Self-supervised speech (SSL) models have recently become widely adopted for
many downstream speech processing tasks. The general usage pattern is to employ
SSL models as feature extractors, and then train a downstream prediction head
to solve a specific task. However, different layers of SSL models have been
shown to capture different types of information, and the methods of combining
them are not well studied. To this end, we extend the general framework for SSL
model utilization by proposing the interface that connects the upstream and
downstream. Under this view, the dominant technique of combining features via a
layerwise weighted sum can be regarded as a specific interface. We propose
several alternative interface designs and demonstrate that the weighted sum
interface is suboptimal for many tasks. In particular, we show that a
convolutional interface whose depth scales logarithmically with the depth of
the upstream model consistently outperforms many other interface designs.
A new interface design for connecting self-supervised speech models to downstream tasks outperforms traditional layerwise weighted sum techniques, particularly when using a convolutional interface with depth scaling logarithmically.
self-supervised speech models feature extractors downstream prediction head layerwise weighted sum convolutional interface