RLVR-World: Training World Models with Reinforcement Learning
World models predict state transitions in response to actions and are
increasingly developed across diverse modalities. However, standard training
objectives such as maximum likelihood estimation (MLE) often misalign with
task-specific goals of world models, i.e., transition prediction metrics like
accuracy or perceptual quality. In this paper, we present RLVR-World, a unified
framework that leverages reinforcement learning with verifiable rewards (RLVR)
to directly optimize world models for such metrics. Despite formulating world
modeling as autoregressive prediction of tokenized sequences, RLVR-World
evaluates metrics of decoded predictions as verifiable rewards. We demonstrate
substantial performance gains on both language- and video-based world models
across domains, including text games, web navigation, and robot manipulation.
Our work indicates that, beyond recent advances in reasoning language models,
RLVR offers a promising post-training paradigm for enhancing the utility of
generative models more broadly.
682e8369b16c79c271b4db81
Jialong Wu
extracted_confirmed
2025-05-22T05:44:24.633Z
643b866bff50448bcfc7d1d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q12v-BatVKimoi8q-coi-.jpeg
Jialong Wu
user
manchery
682e8369b16c79c271b4db82
Shaofeng Yin
682e8369b16c79c271b4db83
Ningya Feng
682e8369b16c79c271b4db84
Mingsheng Long
RLVR-World: Training World Models with Reinforcement Learning
World models predict state transitions in response to actions and are
increasingly developed across diverse modalities. However, standard training
objectives such as maximum likelihood estimation (MLE) often misalign with
task-specific goals of world models, i.e., transition prediction metrics like
accuracy or perceptual quality. In this paper, we present RLVR-World, a unified
framework that leverages reinforcement learning with verifiable rewards (RLVR)
to directly optimize world models for such metrics. Despite formulating world
modeling as autoregressive prediction of tokenized sequences, RLVR-World
evaluates metrics of decoded predictions as verifiable rewards. We demonstrate
substantial performance gains on both language- and video-based world models
across domains, including text games, web navigation, and robot manipulation.
Our work indicates that, beyond recent advances in reasoning language models,
RLVR offers a promising post-training paradigm for enhancing the utility of
generative models more broadly.
RLVR-World uses reinforcement learning with verifiable rewards to optimize world models for task-specific metrics, achieving improved performance across language and video domains.
world models maximum likelihood estimation transition prediction reinforcement learning verifiable rewards autoregressive prediction tokenized sequences text games web navigation robot manipulation