LongAgent: Scaling Language Models to 128k Context through Multi-Agent
  Collaboration
Large language models (LLMs) have demonstrated impressive performance in
understanding language and executing complex reasoning tasks. However, LLMs
with long context windows have been notorious for their expensive training
costs and high inference latency. Even the most advanced models such as GPT-4
and Claude2 often make mistakes when processing inputs of over 100k tokens, a
phenomenon also known as lost in the middle. In this paper, we propose
LongAgent, a method based on multi-agent collaboration, which scales
LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority
in long-text processing compared to GPT-4. In LongAgent, a leader is
responsible for understanding user intent and directing team members to acquire
information from documents. Due to members' hallucinations, it is non-trivial
for a leader to obtain accurate information from the responses of dozens to
hundreds of members. To address this, we develop an inter-member
communication mechanism to resolve response conflicts caused by hallucinations
through information sharing. Our experimental results indicate that
LongAgent offers a promising alternative for long-text processing. The
agent team instantiated with LLaMA-7B achieves significant improvements in
tasks such as 128k-long text retrieval, multi-hop question answering, compared
to GPT-4.
65d42906cf4034c90c00f943
Jun Zhao
65d42906cf4034c90c00f944
Can Zu
65d42906cf4034c90c00f945
Hao Xu
65d42906cf4034c90c00f946
Yi Lu
65d42906cf4034c90c00f947
Wei He
claimed_verified
2024-10-25T09:32:16.410Z
66ecee857264238429a1211f
https://cdn-avatars.huggingface.co/v1/production/uploads/66ecee857264238429a1211f/TbuM7ToLBrSxDF8mOccpK.jpeg
Wei He
user
hewei2001
65d42906cf4034c90c00f948
Yiwen Ding
claimed_verified
2024-06-11T06:59:56.763Z
650819ad95c45596854271a3
/avatars/9517fd62eef104c33884dfb3f64249bf.svg
wen
user
cindywen
65d42906cf4034c90c00f949
Tao Gui
65d42906cf4034c90c00f94a
Qi Zhang
65d42906cf4034c90c00f94b
Xuanjing Huang
LongAgent: Scaling Language Models to 128k Context through Multi-Agent
  Collaboration
Large language models (LLMs) have demonstrated impressive performance in
understanding language and executing complex reasoning tasks. However, LLMs
with long context windows have been notorious for their expensive training
costs and high inference latency. Even the most advanced models such as GPT-4
and Claude2 often make mistakes when processing inputs of over 100k tokens, a
phenomenon also known as lost in the middle. In this paper, we propose
LongAgent, a method based on multi-agent collaboration, which scales
LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority
in long-text processing compared to GPT-4. In LongAgent, a leader is
responsible for understanding user intent and directing team members to acquire
information from documents. Due to members' hallucinations, it is non-trivial
for a leader to obtain accurate information from the responses of dozens to
hundreds of members. To address this, we develop an inter-member
communication mechanism to resolve response conflicts caused by hallucinations
through information sharing. Our experimental results indicate that
LongAgent offers a promising alternative for long-text processing. The
agent team instantiated with LLaMA-7B achieves significant improvements in
tasks such as 128k-long text retrieval, multi-hop question answering, compared
to GPT-4.
LongAgent, a multi-agent collaboration method, scales large language models to handle up to 128K tokens with improved accuracy and performance in long-text processing tasks compared to GPT-4.
multi-agent collaboration context window hallucinations inter-member communication long-text processing LLaMA text retrieval multi-hop question answering