Steering Autoregressive Music Generation with Recursive Feature Machines
Controllable music generation remains a significant challenge, with existing
methods often requiring model retraining or introducing audible artifacts. We
introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs)
to enable fine-grained, interpretable control over frozen, pre-trained music
models by directly steering their internal activations. RFMs analyze a model's
internal gradients to produce interpretable "concept directions", or specific
axes in the activation space that correspond to musical attributes like notes
or chords. We first train lightweight RFM probes to discover these directions
within MusicGen's hidden states; then, during inference, we inject them back
into the model to guide the generation process in real-time without per-step
optimization. We present advanced mechanisms for this control, including
dynamic, time-varying schedules and methods for the simultaneous enforcement of
multiple musical properties. Our method successfully navigates the trade-off
between control and generation quality: we can increase the accuracy of
generating a target musical note from 0.23 to 0.82, while text prompt adherence
remains within approximately 0.02 of the unsteered baseline, demonstrating
effective control with minimal impact on prompt fidelity. We release code to
encourage further exploration on RFMs in the music domain.
68fa75c0f158a71c5a2f56d6
Daniel Zhao
68fa75c0f158a71c5a2f56d7
Daniel Beaglehole
68fa75c0f158a71c5a2f56d8
Taylor Berg-Kirkpatrick
68fa75c0f158a71c5a2f56d9
Julian McAuley
68fa75c0f158a71c5a2f56da
Zachary Novack
claimed_verified
2025-10-24T01:06:59.534Z
643060c6cb3fe707b24c53a2
https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
Zachary Novack
user
ZacharyNovack
Steering Autoregressive Music Generation with Recursive Feature Machines
Controllable music generation remains a significant challenge, with existing
methods often requiring model retraining or introducing audible artifacts. We
introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs)
to enable fine-grained, interpretable control over frozen, pre-trained music
models by directly steering their internal activations. RFMs analyze a model's
internal gradients to produce interpretable "concept directions", or specific
axes in the activation space that correspond to musical attributes like notes
or chords. We first train lightweight RFM probes to discover these directions
within MusicGen's hidden states; then, during inference, we inject them back
into the model to guide the generation process in real-time without per-step
optimization. We present advanced mechanisms for this control, including
dynamic, time-varying schedules and methods for the simultaneous enforcement of
multiple musical properties. Our method successfully navigates the trade-off
between control and generation quality: we can increase the accuracy of
generating a target musical note from 0.23 to 0.82, while text prompt adherence
remains within approximately 0.02 of the unsteered baseline, demonstrating
effective control with minimal impact on prompt fidelity. We release code to
encourage further exploration on RFMs in the music domain.
MusicRFM uses Recursive Feature Machines to enable real-time, fine-grained control over pre-trained music models by steering their internal activations, improving musical note accuracy with minimal impact on prompt fidelity.
Recursive Feature Machines RFMs concept directions activation space MusicGen hidden states dynamic schedules simultaneous enforcement musical properties text prompt adherence