TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and
  Image-to-Video Generation
Video generation has many unique challenges beyond those of image generation.
The temporal dimension introduces extensive possible variations across frames,
over which consistency and continuity may be violated. In this study, we move
beyond evaluating simple actions and argue that generated videos should
incorporate the emergence of new concepts and their relation transitions like
in real-world videos as time progresses. To assess the Temporal
Compositionality of video generation models, we propose TC-Bench, a benchmark
of meticulously crafted text prompts, corresponding ground truth videos, and
robust evaluation metrics. The prompts articulate the initial and final states
of scenes, effectively reducing ambiguities for frame development and
simplifying the assessment of transition completion. In addition, by collecting
aligned real-world videos corresponding to the prompts, we expand TC-Bench's
applicability from text-conditional models to image-conditional ones that can
perform generative frame interpolation. We also develop new metrics to measure
the completeness of component transitions in generated videos, which
demonstrate significantly higher correlations with human judgments than
existing metrics. Our comprehensive experimental results reveal that most video
generators achieve less than 20% of the compositional changes, highlighting
enormous space for future improvement. Our analysis indicates that current
video generation models struggle to interpret descriptions of compositional
changes and synthesize various components across different time steps.
666bbcbcd85fde8ee602af1d
Weixi Feng
claimed_verified
2024-06-14T07:22:19.259Z
6259e9489326c60347dd2b6c
/avatars/6aae6282ddee96b899b7384154e9b0a0.svg
Weixi Feng
user
weixifeng
666bbcbcd85fde8ee602af1e
Jiachen Li
claimed_verified
2024-06-18T07:20:13.124Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
666bbcbcd85fde8ee602af1f
Michael Saxon
claimed_verified
2024-07-08T20:00:58.657Z
6002c1db698168af3bb9f4a5
https://cdn-avatars.huggingface.co/v1/production/uploads/6002c1db698168af3bb9f4a5/IO4sXemgIhKzib7CfeWAC.jpeg
Michael Saxon
user
saxon
666bbcbcd85fde8ee602af20
Tsu-jui Fu
666bbcbcd85fde8ee602af21
Wenhu Chen
666bbcbcd85fde8ee602af22
William Yang Wang
TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and
  Image-to-Video Generation
Video generation has many unique challenges beyond those of image generation.
The temporal dimension introduces extensive possible variations across frames,
over which consistency and continuity may be violated. In this study, we move
beyond evaluating simple actions and argue that generated videos should
incorporate the emergence of new concepts and their relation transitions like
in real-world videos as time progresses. To assess the Temporal
Compositionality of video generation models, we propose TC-Bench, a benchmark
of meticulously crafted text prompts, corresponding ground truth videos, and
robust evaluation metrics. The prompts articulate the initial and final states
of scenes, effectively reducing ambiguities for frame development and
simplifying the assessment of transition completion. In addition, by collecting
aligned real-world videos corresponding to the prompts, we expand TC-Bench's
applicability from text-conditional models to image-conditional ones that can
perform generative frame interpolation. We also develop new metrics to measure
the completeness of component transitions in generated videos, which
demonstrate significantly higher correlations with human judgments than
existing metrics. Our comprehensive experimental results reveal that most video
generators achieve less than 20% of the compositional changes, highlighting
enormous space for future improvement. Our analysis indicates that current
video generation models struggle to interpret descriptions of compositional
changes and synthesize various components across different time steps.
A benchmark named TC-Bench is proposed to evaluate the Temporal Compositionality of video generation models, highlighting significant gaps in current models' ability to handle compositional changes and transitions over time.
TC-Bench Temporal Compositionality text prompts ground truth videos evaluation metrics generative frame interpolation frame development transition completion human judgments compositional changes