From Data to Rewards: a Bilevel Optimization Perspective on Maximum
  Likelihood Estimation
Generative models form the backbone of modern machine learning, underpinning
state-of-the-art systems in text, vision, and multimodal applications. While
Maximum Likelihood Estimation has traditionally served as the dominant training
paradigm, recent work have highlighted its limitations, particularly in
generalization and susceptibility to catastrophic forgetting compared to
Reinforcement Learning techniques, such as Policy Gradient methods. However,
these approaches depend on explicit reward signals, which are often unavailable
in practice, leaving open the fundamental problem of how to align generative
models when only high-quality datasets are accessible. In this work, we address
this challenge via a Bilevel Optimization framework, where the reward function
is treated as the optimization variable of an outer-level problem, while a
policy gradient objective defines the inner-level. We then conduct a
theoretical analysis of this optimization problem in a tractable setting and
extract insights that, as we demonstrate, generalize to applications such as
tabular classification and model-based reinforcement learning. We release the
code at https://github.com/abenechehab/nll_to_po .
68ecc73acd07fb414898cce2
Abdelhakim Benechehab
claimed_verified
2025-10-14T10:23:51.762Z
621d59ebd3df05d67132e8d9
https://cdn-avatars.huggingface.co/v1/production/uploads/621d59ebd3df05d67132e8d9/0gPfPTRKKnz5kq0InTqm5.jpeg
Abdelhakim Benechehab
user
abenechehab
68ecc73acd07fb414898cce3
Gabriel Singer
68ecc73acd07fb414898cce4
Corentin Léger
claimed_verified
2025-10-14T10:23:48.639Z
650569cd0761d34d28c47eb4
/avatars/669b77101c3dcb9d2cc04bfc8170af36.svg
Corentin Léger
user
corentinlger
68ecc73acd07fb414898cce5
Youssef Attia El Hili
claimed_verified
2025-10-20T12:13:57.151Z
6389b24e51b7825b34286e8b
/avatars/3d10ef7ea849348526ef95c04d21ed7a.svg
Youssef Attia El Hili
user
Yattia
68ecc73acd07fb414898cce6
Giuseppe Paolo
claimed_verified
2025-10-14T14:24:22.207Z
65e98cd8e19214e9d151f29e
https://cdn-avatars.huggingface.co/v1/production/uploads/65e98cd8e19214e9d151f29e/XjQzoVgKVzv8AZBWFQnHz.jpeg
Giuseppe Paolo
user
GPaolo
68ecc73acd07fb414898cce7
Albert Thomas
68ecc73acd07fb414898cce8
Maurizio Filippone
68ecc73acd07fb414898cce9
Balázs Kégl
From Data to Rewards: a Bilevel Optimization Perspective on Maximum
  Likelihood Estimation
Generative models form the backbone of modern machine learning, underpinning
state-of-the-art systems in text, vision, and multimodal applications. While
Maximum Likelihood Estimation has traditionally served as the dominant training
paradigm, recent work have highlighted its limitations, particularly in
generalization and susceptibility to catastrophic forgetting compared to
Reinforcement Learning techniques, such as Policy Gradient methods. However,
these approaches depend on explicit reward signals, which are often unavailable
in practice, leaving open the fundamental problem of how to align generative
models when only high-quality datasets are accessible. In this work, we address
this challenge via a Bilevel Optimization framework, where the reward function
is treated as the optimization variable of an outer-level problem, while a
policy gradient objective defines the inner-level. We then conduct a
theoretical analysis of this optimization problem in a tractable setting and
extract insights that, as we demonstrate, generalize to applications such as
tabular classification and model-based reinforcement learning. We release the
code at https://github.com/abenechehab/nll_to_po .
A bilevel optimization framework is used to align generative models with high-quality datasets in the absence of explicit reward signals, with applications in classification and model-based reinforcement learning.
generative models Maximum Likelihood Estimation Reinforcement Learning Policy Gradient methods Bilevel Optimization reward function policy gradient objective tabular classification model-based reinforcement learning