Controlling Multimodal LLMs via Reward-guided Decoding
As Multimodal Large Language Models (MLLMs) gain widespread applicability, it
is becoming increasingly desirable to adapt them for diverse user needs. In
this paper, we study the adaptation of MLLMs through controlled decoding. To
achieve this, we introduce the first method for reward-guided decoding of MLLMs
and demonstrate its application in improving their visual grounding. Our method
involves building reward models for visual grounding and using them to guide
the MLLM's decoding process. Concretely, we build two separate reward models to
independently control the degree of object precision and recall in the model's
output. Our approach enables on-the-fly controllability of an MLLM's inference
process in two ways: first, by giving control over the relative importance of
each reward function during decoding, allowing a user to dynamically trade off
object precision for recall in image captioning tasks; second, by giving
control over the breadth of the search during decoding, allowing the user to
control the trade-off between the amount of test-time compute and the degree of
visual grounding. We evaluate our method on standard object hallucination
benchmarks, showing that it provides significant controllability over MLLM
inference, while consistently outperforming existing hallucination mitigation
methods.
68a2a72ba4caabb4320e6421
Oscar Mañas
claimed_verified
2025-08-18T06:53:56.284Z
6355d5c460c1b72f62693983
https://cdn-avatars.huggingface.co/v1/production/uploads/6355d5c460c1b72f62693983/RA4FJCgPyqMXgTaSvBUGq.jpeg
Oscar Mañas
user
oscmansan
68a2a72ba4caabb4320e6422
Pierluca D'Oro
68a2a72ba4caabb4320e6423
Koustuv Sinha
68a2a72ba4caabb4320e6424
Adriana Romero-Soriano
68a2a72ba4caabb4320e6425
Michal Drozdzal
68a2a72ba4caabb4320e6426
Aishwarya Agrawal
Controlling Multimodal LLMs via Reward-guided Decoding
As Multimodal Large Language Models (MLLMs) gain widespread applicability, it
is becoming increasingly desirable to adapt them for diverse user needs. In
this paper, we study the adaptation of MLLMs through controlled decoding. To
achieve this, we introduce the first method for reward-guided decoding of MLLMs
and demonstrate its application in improving their visual grounding. Our method
involves building reward models for visual grounding and using them to guide
the MLLM's decoding process. Concretely, we build two separate reward models to
independently control the degree of object precision and recall in the model's
output. Our approach enables on-the-fly controllability of an MLLM's inference
process in two ways: first, by giving control over the relative importance of
each reward function during decoding, allowing a user to dynamically trade off
object precision for recall in image captioning tasks; second, by giving
control over the breadth of the search during decoding, allowing the user to
control the trade-off between the amount of test-time compute and the degree of
visual grounding. We evaluate our method on standard object hallucination
benchmarks, showing that it provides significant controllability over MLLM
inference, while consistently outperforming existing hallucination mitigation
methods.
A reward-guided decoding method for Multimodal Large Language Models (MLLMs) improves visual grounding by controlling object precision and recall, offering dynamic trade-offs between compute and grounding quality.
Multimodal Large Language Models MLLMs reward-guided decoding visual grounding reward models object precision object recall image captioning object hallucination hallucination mitigation