Merlin:Empowering Multimodal LLMs with Foresight Minds
Humans possess the remarkable ability to foresee the future to a certain
extent based on present observations, a skill we term as foresight minds.
However, this capability remains largely under explored within existing
Multimodal Large Language Models (MLLMs), hindering their capacity to learn the
fundamental principles of how things operate and the intentions behind the
observed subjects. To address this issue, we introduce the integration of
future modeling into the existing learning frameworks of MLLMs. By utilizing
the subject trajectory, a highly structured representation of a consecutive
frame sequence, as a learning objective, we aim to bridge the gap between the
past and the future. We propose two innovative methods to empower MLLMs with
foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning
(FIT), which are inspired by the modern learning paradigm of LLMs.
Specifically, FPT jointly training various tasks centered on trajectories,
enabling MLLMs to learn how to attend and predict entire trajectories from a
given initial observation. Then, FIT requires MLLMs to first predict
trajectories of related objects and then reason about potential future events
based on them. Aided by FPT and FIT, we build a novel and unified MLLM named
Merlin that supports multi-images input and analysis about potential actions of
multiple objects for the future reasoning. Experimental results show Merlin
powerful foresight minds with impressive performance on both future reasoning
and visual comprehension tasks.
656d34623dbac3a83d3a19f0
En Yu
656d34623dbac3a83d3a19f1
Liang Zhao
656d34623dbac3a83d3a19f2
Yana Wei
admin_assigned
2023-12-04T08:10:56.153Z
64ae4d62179421d320b67c26
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae4d62179421d320b67c26/nz-tY6hX7mcDzhdtBmG8K.jpeg
Yana Wei
user
llwswyn
656d34623dbac3a83d3a19f3
Jinrong Yang
admin_assigned
2023-12-04T08:11:33.191Z
650bd10c1f2949b99e65ef23
/avatars/8f36f844690b64bcc7c358063f2bdc3b.svg
Jinrong Yang
user
YancieYJR
656d34623dbac3a83d3a19f4
Dongming Wu
656d34623dbac3a83d3a19f5
Lingyu Kong
claimed_verified
2024-04-18T08:11:24.894Z
650d23b69169ea733178f508
/avatars/5f9a7ab6b6a44e74c3eb6a59da2e21ed.svg
Lingyu Kong
user
kppkkp
656d34623dbac3a83d3a19f6
Haoran Wei
admin_assigned
2023-12-04T08:13:52.338Z
6436618aeef1f55654a9f458
https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
Haoran Wei
user
HaoranWei
656d34623dbac3a83d3a19f7
Tiancai Wang
656d34623dbac3a83d3a19f8
Zheng Ge
656d34623dbac3a83d3a19f9
Xiangyu Zhang
656d34623dbac3a83d3a19fa
Wenbing Tao
Merlin:Empowering Multimodal LLMs with Foresight Minds
Humans possess the remarkable ability to foresee the future to a certain
extent based on present observations, a skill we term as foresight minds.
However, this capability remains largely under explored within existing
Multimodal Large Language Models (MLLMs), hindering their capacity to learn the
fundamental principles of how things operate and the intentions behind the
observed subjects. To address this issue, we introduce the integration of
future modeling into the existing learning frameworks of MLLMs. By utilizing
the subject trajectory, a highly structured representation of a consecutive
frame sequence, as a learning objective, we aim to bridge the gap between the
past and the future. We propose two innovative methods to empower MLLMs with
foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning
(FIT), which are inspired by the modern learning paradigm of LLMs.
Specifically, FPT jointly training various tasks centered on trajectories,
enabling MLLMs to learn how to attend and predict entire trajectories from a
given initial observation. Then, FIT requires MLLMs to first predict
trajectories of related objects and then reason about potential future events
based on them. Aided by FPT and FIT, we build a novel and unified MLLM named
Merlin that supports multi-images input and analysis about potential actions of
multiple objects for the future reasoning. Experimental results show Merlin
powerful foresight minds with impressive performance on both future reasoning
and visual comprehension tasks.
A novel MLLM, Merlin, is introduced which incorporates future modeling through FPT and FIT, enhancing foresight and future reasoning capabilities.
Multimodal Large Language Models MLLMs foresight minds future modeling subject trajectory Foresight Pre-Training FPT Foresight Instruction-Tuning FIT future reasoning visual comprehension