Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
  Tasks in Superposition
Large Language Models (LLMs) have demonstrated remarkable in-context learning
(ICL) capabilities. In this study, we explore a surprising phenomenon related
to ICL: LLMs can perform multiple, computationally distinct ICL tasks
simultaneously, during a single inference call, a capability we term "task
superposition". We provide empirical evidence of this phenomenon across various
LLM families and scales and show that this phenomenon emerges even if we train
the model to in-context learn one task at a time. We offer theoretical
explanations that this capability is well within the expressive power of
transformers. We also explore how LLMs internally compose task vectors during
superposition. Furthermore, we show that larger models can solve more ICL tasks
in parallel, and better calibrate their output distribution. Our findings offer
insights into the latent capabilities of LLMs, further substantiate the
perspective of "LLMs as superposition of simulators", and raise questions about
the mechanisms enabling simultaneous task execution.
67093c6d10d26dfd98500536
Zheyang Xiong
claimed_verified
2024-10-17T11:27:51.579Z
67084dbb652deb3c09d8f633
/avatars/6489cc414ec6b7be3523e335ba14888f.svg
Zheyang Xiong
user
zheyang-hf
67093c6d10d26dfd98500537
Ziyang Cai
67093c6d10d26dfd98500538
John Cooper
67093c6d10d26dfd98500539
Albert Ge
claimed_verified
2024-11-15T09:43:29.776Z
650263c89a612aa33a018383
/avatars/a1415a4696da98ffebc1ac798cbab052.svg
Albert Ge
user
albertge
67093c6d10d26dfd9850053a
Vasilis Papageorgiou
claimed_verified
2024-10-13T20:16:05.070Z
631bd99e9b7736c65a62287c
https://cdn-avatars.huggingface.co/v1/production/uploads/1662769518341-noauth.jpeg
Vasilis Papageorgiou
user
vpapageorgiou
67093c6d10d26dfd9850053b
Zack Sifakis
67093c6d10d26dfd9850053c
Angeliki Giannou
67093c6d10d26dfd9850053d
Ziqian Lin
67093c6d10d26dfd9850053e
Liu Yang
67093c6d10d26dfd9850053f
Saurabh Agarwal
67093c6d10d26dfd98500540
Grigorios G Chrysos
67093c6d10d26dfd98500541
Samet Oymak
67093c6d10d26dfd98500542
Kangwook Lee
67093c6d10d26dfd98500543
Dimitris Papailiopoulos
extracted_pending
2024-10-11T14:55:42.107Z
64428cf07feb866811b0be6a
/avatars/f30d339a02ce070c7e77315b59e20359.svg
Dimitris Papailiopoulos
user
anadim
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
  Tasks in Superposition
Large Language Models (LLMs) have demonstrated remarkable in-context learning
(ICL) capabilities. In this study, we explore a surprising phenomenon related
to ICL: LLMs can perform multiple, computationally distinct ICL tasks
simultaneously, during a single inference call, a capability we term "task
superposition". We provide empirical evidence of this phenomenon across various
LLM families and scales and show that this phenomenon emerges even if we train
the model to in-context learn one task at a time. We offer theoretical
explanations that this capability is well within the expressive power of
transformers. We also explore how LLMs internally compose task vectors during
superposition. Furthermore, we show that larger models can solve more ICL tasks
in parallel, and better calibrate their output distribution. Our findings offer
insights into the latent capabilities of LLMs, further substantiate the
perspective of "LLMs as superposition of simulators", and raise questions about
the mechanisms enabling simultaneous task execution.
LLMs can perform multiple distinct in-context learning tasks simultaneously during a single inference, a phenomenon termed "task superposition," which scales with model size and improves output calibration.
Large Language Models in-context learning task superposition transformers task vectors output distribution calibration