Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models
The new paradigm of test-time scaling has yielded remarkable breakthroughs in
Large Language Models (LLMs) (e.g. reasoning models) and in generative vision
models, allowing models to allocate additional computation during inference to
effectively tackle increasingly complex problems. Despite the improvements of
this approach, an important limitation emerges: the substantial increase in
computation time makes the process slow and impractical for many applications.
Given the success of this paradigm and its growing usage, we seek to preserve
its benefits while eschewing the inference overhead. In this work we propose
one solution to the critical problem of integrating test-time scaling knowledge
into a model during post-training. Specifically, we replace reward guided
test-time noise optimization in diffusion models with a Noise Hypernetwork that
modulates initial input noise. We propose a theoretically grounded framework
for learning this reward-tilted distribution for distilled generators, through
a tractable noise-space objective that maintains fidelity to the base model
while optimizing for desired characteristics. We show that our approach
recovers a substantial portion of the quality gains from explicit test-time
optimization at a fraction of the computational cost. Code is available at
https://github.com/ExplainableML/HyperNoise
689d5ebbb083e610d741ea45
Luca Eyring
claimed_verified
2025-08-18T06:59:07.512Z
64edae322d5805cb998c7047
/avatars/6f1e887842d339a6c80b990491da1950.svg
Luca Eyring
user
lucaeyring
689d5ebbb083e610d741ea46
Shyamgopal Karthik
claimed_verified
2025-08-14T13:37:25.592Z
6254599b6e36fe62e141c8f9
/avatars/08d6b68b92c7bfd0a8be022ba9f2f289.svg
Shyamgopal Karthik
user
shyamgopal
689d5ebbb083e610d741ea47
Alexey Dosovitskiy
689d5ebbb083e610d741ea48
Nataniel Ruiz
claimed_verified
2025-08-20T19:13:35.230Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
689d5ebbb083e610d741ea49
Zeynep Akata
Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models
The new paradigm of test-time scaling has yielded remarkable breakthroughs in
Large Language Models (LLMs) (e.g. reasoning models) and in generative vision
models, allowing models to allocate additional computation during inference to
effectively tackle increasingly complex problems. Despite the improvements of
this approach, an important limitation emerges: the substantial increase in
computation time makes the process slow and impractical for many applications.
Given the success of this paradigm and its growing usage, we seek to preserve
its benefits while eschewing the inference overhead. In this work we propose
one solution to the critical problem of integrating test-time scaling knowledge
into a model during post-training. Specifically, we replace reward guided
test-time noise optimization in diffusion models with a Noise Hypernetwork that
modulates initial input noise. We propose a theoretically grounded framework
for learning this reward-tilted distribution for distilled generators, through
a tractable noise-space objective that maintains fidelity to the base model
while optimizing for desired characteristics. We show that our approach
recovers a substantial portion of the quality gains from explicit test-time
optimization at a fraction of the computational cost. Code is available at
https://github.com/ExplainableML/HyperNoise
A Noise Hypernetwork is introduced to integrate test-time scaling knowledge into diffusion models, reducing computational cost while maintaining quality.
test-time scaling Large Language Models generative vision models diffusion models reward guided test-time noise optimization Noise Hypernetwork noise-space objective distilled generators