AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
We present Any-Modality Augmented Language Model (AnyMAL), a unified model
that reasons over diverse input modality signals (i.e. text, image, video,
audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
powerful text-based reasoning abilities of the state-of-the-art LLMs including
LLaMA-2 (70B), and converts modality-specific signals to the joint textual
space through a pre-trained aligner module. To further strengthen the
multimodal LLM's capabilities, we fine-tune the model with a multimodal
instruction set manually collected to cover diverse topics and tasks beyond
simple QAs. We conduct comprehensive empirical analysis comprising both human
and automatic evaluations, and demonstrate state-of-the-art performance on
various multimodal tasks.
65162bcf3e5a12e0ceb42dfe
Seungwhan Moon
admin_assigned
2023-09-29T11:18:51.029Z
64c9ae24e761f47061448bfb
/avatars/8835bcf769fd738e8c66fa7a52ddc421.svg
Seungwhan Moon
user
shanemoon27
65162bcf3e5a12e0ceb42dff
Andrea Madotto
admin_assigned
2023-09-29T11:18:58.812Z
609bc5719da7ce3ca8930b8e
https://cdn-avatars.huggingface.co/v1/production/uploads/1620821302901-noauth.jpeg
Andrea Madotto
user
andreamad8
65162bcf3e5a12e0ceb42e00
Zhaojiang Lin
admin_assigned
2023-09-29T11:19:05.658Z
64b6d095dbbd1f2cdb62a946
/avatars/b5f97735045d05ba5d30ee1fbf3f9bae.svg
Zhaojiang Lin
user
zlinao
65162bcf3e5a12e0ceb42e01
Tushar Nagarajan
65162bcf3e5a12e0ceb42e02
Matt Smith
claimed_verified
2023-10-04T07:37:45.913Z
630872613485f377829de16f
/avatars/412d46da44f357b549d58e82d789d98d.svg
Matt Smith
user
mjksmith
65162bcf3e5a12e0ceb42e03
Shashank Jain
admin_assigned
2023-09-29T11:20:04.248Z
64b6bdc461dc301f13230172
/avatars/f43bb430c1c5508c6b52bdd9c0a8bfae.svg
Shashank Jain
user
shashankneocfc
65162bcf3e5a12e0ceb42e04
Chun-Fu Yeh
admin_assigned
2023-09-29T11:20:13.048Z
64110236230ce11b1be5f869
/avatars/f8718d21f7ea8b908afa938233a77f83.svg
Chun-Fu Yeh
user
cfyeh
65162bcf3e5a12e0ceb42e05
Prakash Murugesan
claimed_verified
2023-11-10T07:58:25.576Z
654dd8a10011607c3125f107
/avatars/9235811dd394c08116ceed32d943e968.svg
Prakash Murugesan
user
prakash14
65162bcf3e5a12e0ceb42e06
Peyman Heidari
65162bcf3e5a12e0ceb42e07
Yue Liu
claimed_verified
2024-06-10T17:09:07.936Z
66671b27ad9cd05ea74202a5
/avatars/a6fa26c173f3e5b0748e5fb007c3105e.svg
Yue Liu
user
liu-yue
65162bcf3e5a12e0ceb42e08
Kavya Srinet
claimed_verified
2023-12-04T08:03:36.763Z
656a27e43eb5f0b6a9bef54a
/avatars/32176455b54496917152b009e03efc2a.svg
Kavya Srinet
user
kavyasrinet
65162bcf3e5a12e0ceb42e09
Babak Damavandi
65162bcf3e5a12e0ceb42e0a
Anuj Kumar
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
We present Any-Modality Augmented Language Model (AnyMAL), a unified model
that reasons over diverse input modality signals (i.e. text, image, video,
audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
powerful text-based reasoning abilities of the state-of-the-art LLMs including
LLaMA-2 (70B), and converts modality-specific signals to the joint textual
space through a pre-trained aligner module. To further strengthen the
multimodal LLM's capabilities, we fine-tune the model with a multimodal
instruction set manually collected to cover diverse topics and tasks beyond
simple QAs. We conduct comprehensive empirical analysis comprising both human
and automatic evaluations, and demonstrate state-of-the-art performance on
various multimodal tasks.
AnyMAL is a unified model that processes multiple modalities (text, image, video, audio, IMU) and generates text, achieving top performance in multimodal tasks through a pre-trained aligner and fine-tuning with diverse instructions.
Any-Modality Augmented Language Model AnyMAL unified model multimodal input modality signals LLMs LLaMA-2 pre-trained aligner module multimodal instruction set empirical analysis human evaluations automatic evaluations