World Model on Million-Length Video And Language With RingAttention
Current language models fall short in understanding aspects of the world not
easily described in words, and struggle with complex, long-form tasks. Video
sequences offer valuable temporal information absent in language and static
images, making them attractive for joint modeling with language. Such models
could develop a understanding of both human textual knowledge and the physical
world, enabling broader AI capabilities for assisting humans. However, learning
from millions of tokens of video and language sequences poses challenges due to
memory constraints, computational complexity, and limited datasets. To address
these challenges, we curate a large dataset of diverse videos and books,
utilize the RingAttention technique to scalably train on long sequences, and
gradually increase context size from 4K to 1M tokens. This paper makes the
following contributions: (a) Largest context size neural network: We train one
of the largest context size transformers on long video and language sequences,
setting new benchmarks in difficult retrieval tasks and long video
understanding. (b) Solutions for overcoming vision-language training
challenges, including using masked sequence packing for mixing different
sequence lengths, loss weighting to balance language and vision, and
model-generated QA dataset for long sequence chat. (c) A highly-optimized
implementation with RingAttention, masked sequence packing, and other key
features for training on millions-length multimodal sequences. (d) Fully
open-sourced a family of 7B parameter models capable of processing long text
documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M
tokens. This work paves the way for training on massive datasets of long video
and language to develop understanding of both human knowledge and the
multimodal world, and broader capabilities.
65cc2e8f9a4ee02bbbbf78e5
Hao Liu
admin_assigned
2024-02-14T10:52:25.419Z
62aa53fc5df2251f7df798c6
/avatars/1c7828ce861c30ddd0afceff355a92ec.svg
Hao
user
haoliu
65cc2e8f9a4ee02bbbbf78e6
Wilson Yan
extracted_pending
2024-02-14T03:08:00.254Z
63443747bcabbbc21a4c4aae
/avatars/822dcdd9d94a79f04073ef64df0a0a9e.svg
Wilson
user
wilson1yan
65cc2e8f9a4ee02bbbbf78e7
Matei Zaharia
admin_assigned
2024-02-14T10:50:28.065Z
642509c6d476e4ad5568dd31
https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
Matei Zaharia
user
mateiz
65cc2e8f9a4ee02bbbbf78e8
Pieter Abbeel
World Model on Million-Length Video And Language With RingAttention
Current language models fall short in understanding aspects of the world not
easily described in words, and struggle with complex, long-form tasks. Video
sequences offer valuable temporal information absent in language and static
images, making them attractive for joint modeling with language. Such models
could develop a understanding of both human textual knowledge and the physical
world, enabling broader AI capabilities for assisting humans. However, learning
from millions of tokens of video and language sequences poses challenges due to
memory constraints, computational complexity, and limited datasets. To address
these challenges, we curate a large dataset of diverse videos and books,
utilize the RingAttention technique to scalably train on long sequences, and
gradually increase context size from 4K to 1M tokens. This paper makes the
following contributions: (a) Largest context size neural network: We train one
of the largest context size transformers on long video and language sequences,
setting new benchmarks in difficult retrieval tasks and long video
understanding. (b) Solutions for overcoming vision-language training
challenges, including using masked sequence packing for mixing different
sequence lengths, loss weighting to balance language and vision, and
model-generated QA dataset for long sequence chat. (c) A highly-optimized
implementation with RingAttention, masked sequence packing, and other key
features for training on millions-length multimodal sequences. (d) Fully
open-sourced a family of 7B parameter models capable of processing long text
documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M
tokens. This work paves the way for training on massive datasets of long video
and language to develop understanding of both human knowledge and the
multimodal world, and broader capabilities.
Large-scale training of a transformer model for long video and language sequences using RingAttention and masked sequence packing improves performance on retrieval and long-form understanding tasks.
RingAttention masked sequence packing long video large context size transformers multimodal sequences loss weighting model-generated QA dataset