InstanceCap: Improving Text-to-Video Generation via Instance-aware
  Structured Caption
Text-to-video generation has evolved rapidly in recent years, delivering
remarkable results. Training typically relies on video-caption paired data,
which plays a crucial role in enhancing generation performance. However,
current video captions often suffer from insufficient details, hallucinations
and imprecise motion depiction, affecting the fidelity and consistency of
generated videos. In this work, we propose a novel instance-aware structured
caption framework, termed InstanceCap, to achieve instance-level and
fine-grained video caption for the first time. Based on this scheme, we design
an auxiliary models cluster to convert original video into instances to enhance
instance fidelity. Video instances are further used to refine dense prompts
into structured phrases, achieving concise yet precise descriptions.
Furthermore, a 22K InstanceVid dataset is curated for training, and an
enhancement pipeline that tailored to InstanceCap structure is proposed for
inference. Experimental results demonstrate that our proposed InstanceCap
significantly outperform previous models, ensuring high fidelity between
captions and videos while reducing hallucinations.
675ba7108e73e58d6574019e
Tiehan Fan
claimed_verified
2024-12-13T15:22:20.639Z
65927f3b754092f6b1e187a7
https://cdn-avatars.huggingface.co/v1/production/uploads/65927f3b754092f6b1e187a7/gUrNvIQHmsl1vLwSUxpmL.jpeg
tiehan fan
user
AnonMegumi
675ba7108e73e58d6574019f
Kepan Nan
admin_assigned
2024-12-16T10:05:10.506Z
66271e50f82ffd771a2720ee
/avatars/34f0f5f954a8c8ff41f86320813b034a.svg
nankepan
user
nankepan
675ba7108e73e58d657401a0
Rui Xie
claimed_verified
2024-12-16T09:42:20.696Z
660a7ecf14cfe4973e0acfe1
/avatars/e488058397f2b7a617515a4f721a9a00.svg
Rui Xie
user
SherryX
675ba7108e73e58d657401a1
Penghao Zhou
675ba7108e73e58d657401a2
Zhenheng Yang
admin_assigned
2024-12-16T10:05:24.220Z
6421183b69a2c2933882d652
/avatars/66813a8fa22915087cccd4dbfb945ca7.svg
Zhenheng Yang
user
zhenheny
675ba7108e73e58d657401a3
Chaoyou Fu
admin_assigned
2024-12-16T10:06:26.985Z
653b268cd1041ca9188954da
/avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
BradyFU
user
BradyFU
675ba7108e73e58d657401a4
Xiang Li
675ba7108e73e58d657401a5
Jian Yang
675ba7108e73e58d657401a6
Ying Tai
claimed_verified
2024-12-16T09:42:22.452Z
65734004769f3ee9bde1af10
/avatars/d6310ed861972fd691687d8f47413f33.svg
Ying Tai
user
yingtai
InstanceCap: Improving Text-to-Video Generation via Instance-aware
  Structured Caption
Text-to-video generation has evolved rapidly in recent years, delivering
remarkable results. Training typically relies on video-caption paired data,
which plays a crucial role in enhancing generation performance. However,
current video captions often suffer from insufficient details, hallucinations
and imprecise motion depiction, affecting the fidelity and consistency of
generated videos. In this work, we propose a novel instance-aware structured
caption framework, termed InstanceCap, to achieve instance-level and
fine-grained video caption for the first time. Based on this scheme, we design
an auxiliary models cluster to convert original video into instances to enhance
instance fidelity. Video instances are further used to refine dense prompts
into structured phrases, achieving concise yet precise descriptions.
Furthermore, a 22K InstanceVid dataset is curated for training, and an
enhancement pipeline that tailored to InstanceCap structure is proposed for
inference. Experimental results demonstrate that our proposed InstanceCap
significantly outperform previous models, ensuring high fidelity between
captions and videos while reducing hallucinations.
A novel structured caption framework improves video generation fidelity by converting videos into instance-level descriptions, leading to more precise and hallucination-free captions.
text-to-video generation video-caption paired data instance-aware structured caption instance fidelity dense prompts structured phrases InstanceCap 22K InstanceVid dataset