Large Language Models Implicitly Learn to See and Hear Just By Reading
This paper presents a fascinating find: By training an auto-regressive LLM
model on text tokens, the text model inherently develops internally an ability
to understand images and audio, thereby developing the ability to see and hear
just by reading. Popular audio and visual LLM models fine-tune text LLM models
to give text output conditioned on images and audio embeddings. On the other
hand, our architecture takes in patches of images, audio waveforms or tokens as
input. It gives us the embeddings or category labels typical of a
classification pipeline. We show the generality of text weights in aiding audio
classification for datasets FSD-50K and GTZAN. Further, we show this working
for image classification on CIFAR-10 and Fashion-MNIST, as well on image
patches. This pushes the notion of text-LLMs learning powerful internal
circuits that can be utilized by activating necessary connections for various
applications rather than training models from scratch every single time.
6833cb25fe87d9433dfd2b1c
Prateek Verma
6833cb25fe87d9433dfd2b1d
Mert Pilanci
Large Language Models Implicitly Learn to See and Hear Just By Reading
This paper presents a fascinating find: By training an auto-regressive LLM
model on text tokens, the text model inherently develops internally an ability
to understand images and audio, thereby developing the ability to see and hear
just by reading. Popular audio and visual LLM models fine-tune text LLM models
to give text output conditioned on images and audio embeddings. On the other
hand, our architecture takes in patches of images, audio waveforms or tokens as
input. It gives us the embeddings or category labels typical of a
classification pipeline. We show the generality of text weights in aiding audio
classification for datasets FSD-50K and GTZAN. Further, we show this working
for image classification on CIFAR-10 and Fashion-MNIST, as well on image
patches. This pushes the notion of text-LLMs learning powerful internal
circuits that can be utilized by activating necessary connections for various
applications rather than training models from scratch every single time.
Auto-regressive text LLMs trained on text can develop internal capabilities for understanding images and audio, enabling them to perform classification tasks across different modalities without fine-tuning.
auto-regressive LLM text tokens audio visual embeddings category labels classification FSD-50K GTZAN CIFAR-10 Fashion-MNIST image patches