Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders
Neural audio codecs and autoencoders have emerged as versatile models for
audio compression, transmission, feature-extraction, and latent-space
generation. However, a key limitation is that most are trained to maximize
reconstruction fidelity, often neglecting the specific latent structure
necessary for optimal performance in diverse downstream applications. We
propose a simple, post-hoc framework to address this by modifying the
bottleneck of a pre-trained autoencoder. Our method introduces a
"Re-Bottleneck", an inner bottleneck trained exclusively through latent space
losses to instill user-defined structure. We demonstrate the framework's
effectiveness in three experiments. First, we enforce an ordering on latent
channels without sacrificing reconstruction quality. Second, we align latents
with semantic embeddings, analyzing the impact on downstream diffusion
modeling. Third, we introduce equivariance, ensuring that a filtering operation
on the input waveform directly corresponds to a specific transformation in the
latent space. Ultimately, our Re-Bottleneck framework offers a flexible and
efficient way to tailor representations of neural audio models, enabling them
to seamlessly meet the varied demands of different applications with minimal
additional training.
6871281ec8391850d60979dd
Dimitrios Bralios
6871281ec8391850d60979de
Jonah Casebeer
6871281ec8391850d60979df
Paris Smaragdis
Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders
Neural audio codecs and autoencoders have emerged as versatile models for
audio compression, transmission, feature-extraction, and latent-space
generation. However, a key limitation is that most are trained to maximize
reconstruction fidelity, often neglecting the specific latent structure
necessary for optimal performance in diverse downstream applications. We
propose a simple, post-hoc framework to address this by modifying the
bottleneck of a pre-trained autoencoder. Our method introduces a
"Re-Bottleneck", an inner bottleneck trained exclusively through latent space
losses to instill user-defined structure. We demonstrate the framework's
effectiveness in three experiments. First, we enforce an ordering on latent
channels without sacrificing reconstruction quality. Second, we align latents
with semantic embeddings, analyzing the impact on downstream diffusion
modeling. Third, we introduce equivariance, ensuring that a filtering operation
on the input waveform directly corresponds to a specific transformation in the
latent space. Ultimately, our Re-Bottleneck framework offers a flexible and
efficient way to tailor representations of neural audio models, enabling them
to seamlessly meet the varied demands of different applications with minimal
additional training.
A Re-Bottleneck framework modifies pre-trained autoencoders to introduce user-defined latent structure, enhancing performance in diverse downstream applications without sacrificing reconstruction quality.
neural audio codecs autoencoders latent space Re-Bottleneck latent space losses latent channels semantic embeddings diffusion modeling equivariance latent space transformation