How Confident are Video Models? Empowering Video Models to Express their
  Uncertainty
Generative video models demonstrate impressive text-to-video capabilities,
spurring widespread adoption in many real-world applications. However, like
large language models (LLMs), video generation models tend to hallucinate,
producing plausible videos even when they are factually wrong. Although
uncertainty quantification (UQ) of LLMs has been extensively studied in prior
work, no UQ method for video models exists, raising critical safety concerns.
To our knowledge, this paper represents the first work towards quantifying the
uncertainty of video models. We present a framework for uncertainty
quantification of generative video models, consisting of: (i) a metric for
evaluating the calibration of video models based on robust rank correlation
estimation with no stringent modeling assumptions; (ii) a black-box UQ method
for video models (termed S-QUBED), which leverages latent modeling to
rigorously decompose predictive uncertainty into its aleatoric and epistemic
components; and (iii) a UQ dataset to facilitate benchmarking calibration in
video models. By conditioning the generation task in the latent space, we
disentangle uncertainty arising due to vague task specifications from that
arising from lack of knowledge. Through extensive experiments on benchmark
video datasets, we demonstrate that S-QUBED computes calibrated total
uncertainty estimates that are negatively correlated with the task accuracy and
effectively computes the aleatoric and epistemic constituents.
68e31f7073e20ab577841fb6
Zhiting Mei
68e31f7073e20ab577841fb7
Ola Shorinwa
68e31f7073e20ab577841fb8
Anirudha Majumdar
How Confident are Video Models? Empowering Video Models to Express their
  Uncertainty
Generative video models demonstrate impressive text-to-video capabilities,
spurring widespread adoption in many real-world applications. However, like
large language models (LLMs), video generation models tend to hallucinate,
producing plausible videos even when they are factually wrong. Although
uncertainty quantification (UQ) of LLMs has been extensively studied in prior
work, no UQ method for video models exists, raising critical safety concerns.
To our knowledge, this paper represents the first work towards quantifying the
uncertainty of video models. We present a framework for uncertainty
quantification of generative video models, consisting of: (i) a metric for
evaluating the calibration of video models based on robust rank correlation
estimation with no stringent modeling assumptions; (ii) a black-box UQ method
for video models (termed S-QUBED), which leverages latent modeling to
rigorously decompose predictive uncertainty into its aleatoric and epistemic
components; and (iii) a UQ dataset to facilitate benchmarking calibration in
video models. By conditioning the generation task in the latent space, we
disentangle uncertainty arising due to vague task specifications from that
arising from lack of knowledge. Through extensive experiments on benchmark
video datasets, we demonstrate that S-QUBED computes calibrated total
uncertainty estimates that are negatively correlated with the task accuracy and
effectively computes the aleatoric and epistemic constituents.
A framework for uncertainty quantification in generative video models is introduced, including a metric for calibration, a black-box method called S-QUBED, and a benchmark dataset, demonstrating improved uncertainty estimates and task accuracy.
generative video models text-to-video uncertainty quantification UQ large language models LLMs robust rank correlation estimation latent modeling aleatoric uncertainty epistemic uncertainty latent space benchmark video datasets