Self-alignment of Large Video Language Models with Refined Regularized
  Preference Optimization
Despite recent advances in Large Video Language Models (LVLMs), they still
struggle with fine-grained temporal understanding, hallucinate, and often make
simple mistakes on even simple video question-answering tasks, all of which
pose significant challenges to their safe and reliable deployment in real-world
applications. To address these limitations, we propose a self-alignment
framework that enables LVLMs to learn from their own errors. Our proposed
framework first obtains a training set of preferred and non-preferred response
pairs, where non-preferred responses are generated by incorporating common
error patterns that often occur due to inadequate spatio-temporal
understanding, spurious correlations between co-occurring concepts, and
over-reliance on linguistic cues while neglecting the vision modality, among
others. To facilitate self-alignment of LVLMs with the constructed preferred
and non-preferred response pairs, we introduce Refined Regularized Preference
Optimization (RRPO), a novel preference optimization method that utilizes
sub-sequence-level refined rewards and token-wise KL regularization to address
the limitations of Direct Preference Optimization (DPO). We demonstrate that
RRPO achieves more precise alignment and more stable training compared to DPO.
Our experiments and analysis validate the effectiveness of our approach across
diverse video tasks, including video hallucination, short- and long-video
understanding, and fine-grained temporal reasoning.
68005c2d8da26a09080784c1
Pritam Sarkar
extracted_confirmed
2025-04-17T01:59:05.956Z
6483b3d52193a1768c00c5ff
https://cdn-avatars.huggingface.co/v1/production/uploads/6483b3d52193a1768c00c5ff/uZ212A-sgMr5nAEvlEgHq.jpeg
Pritam Sarkar
user
pritamqu
68005c2d8da26a09080784c2
Ali Etemad
Self-alignment of Large Video Language Models with Refined Regularized
  Preference Optimization
Despite recent advances in Large Video Language Models (LVLMs), they still
struggle with fine-grained temporal understanding, hallucinate, and often make
simple mistakes on even simple video question-answering tasks, all of which
pose significant challenges to their safe and reliable deployment in real-world
applications. To address these limitations, we propose a self-alignment
framework that enables LVLMs to learn from their own errors. Our proposed
framework first obtains a training set of preferred and non-preferred response
pairs, where non-preferred responses are generated by incorporating common
error patterns that often occur due to inadequate spatio-temporal
understanding, spurious correlations between co-occurring concepts, and
over-reliance on linguistic cues while neglecting the vision modality, among
others. To facilitate self-alignment of LVLMs with the constructed preferred
and non-preferred response pairs, we introduce Refined Regularized Preference
Optimization (RRPO), a novel preference optimization method that utilizes
sub-sequence-level refined rewards and token-wise KL regularization to address
the limitations of Direct Preference Optimization (DPO). We demonstrate that
RRPO achieves more precise alignment and more stable training compared to DPO.
Our experiments and analysis validate the effectiveness of our approach across
diverse video tasks, including video hallucination, short- and long-video
understanding, and fine-grained temporal reasoning.
A self-alignment framework using Refined Regularized Preference Optimization helps Large Video Language Models improve accuracy and stability in video question-answering tasks.
Large Video Language Models self-alignment framework preferred and non-preferred response pairs inadequate spatio-temporal understanding spurious correlations linguistic cues Refined Regularized Preference Optimization sub-sequence-level refined rewards token-wise KL regularization Direct Preference Optimization video hallucination short-video understanding long-video understanding fine-grained temporal reasoning