Vector-ICL: In-context Learning with Continuous Vector Representations
Large language models (LLMs) have shown remarkable in-context learning (ICL)
capabilities on textual data. We explore whether these capabilities can be
extended to continuous vectors from diverse domains, obtained from black-box
pretrained encoders. By aligning input data with an LLM's embedding space
through lightweight projectors, we observe that LLMs can effectively process
and learn from these projected vectors, which we term Vector-ICL. In
particular, we find that pretraining projectors with general language modeling
objectives enables Vector-ICL, while task-specific finetuning further enhances
performance. In our experiments across various tasks and modalities, including
text reconstruction, numerical function regression, text classification,
summarization, molecule captioning, time-series classification, graph
classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL
and domain-specific model or tuning. We further conduct analyses and case
studies, indicating the potential of LLMs to process vector representations
beyond traditional token-based paradigms.
670961497a1bf6920b749308
Yufan Zhuang
claimed_verified
2024-10-13T20:16:02.444Z
6438ccbb3b46237de3d052e8
/avatars/baa624d417b0b905e82127dc66346478.svg
Yufan Zhuang
user
yzhuang
670961497a1bf6920b749309
Chandan Singh
670961497a1bf6920b74930a
Liyuan Liu
670961497a1bf6920b74930b
Jingbo Shang
claimed_verified
2025-02-10T09:51:32.459Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
670961497a1bf6920b74930c
Jianfeng Gao
Vector-ICL: In-context Learning with Continuous Vector Representations
Large language models (LLMs) have shown remarkable in-context learning (ICL)
capabilities on textual data. We explore whether these capabilities can be
extended to continuous vectors from diverse domains, obtained from black-box
pretrained encoders. By aligning input data with an LLM's embedding space
through lightweight projectors, we observe that LLMs can effectively process
and learn from these projected vectors, which we term Vector-ICL. In
particular, we find that pretraining projectors with general language modeling
objectives enables Vector-ICL, while task-specific finetuning further enhances
performance. In our experiments across various tasks and modalities, including
text reconstruction, numerical function regression, text classification,
summarization, molecule captioning, time-series classification, graph
classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL
and domain-specific model or tuning. We further conduct analyses and case
studies, indicating the potential of LLMs to process vector representations
beyond traditional token-based paradigms.
LLMs exhibit strong in-context learning capabilities with continuous vector inputs when fine-tuned with pretraining projectors, outperforming traditional methods across diverse tasks and modalities.
large language models in-context learning continuous vectors black-box pretrained encoders lightweight projectors embedding space general language modeling task-specific finetuning vector-ICL text reconstruction numerical function regression text classification summarization molecule captioning time-series classification graph classification fMRI decoding token-based paradigms