Large Language Models Can Self-Improve in Long-context Reasoning
Large language models (LLMs) have achieved substantial progress in processing
long contexts but still struggle with long-context reasoning. Existing
approaches typically involve fine-tuning LLMs with synthetic data, which
depends on annotations from human experts or advanced models like GPT-4, thus
restricting further advancements. To address this issue, we investigate the
potential for LLMs to self-improve in long-context reasoning and propose \ours,
an approach specifically designed for this purpose. This approach is
straightforward: we sample multiple outputs for each question, score them with
Minimum Bayes Risk, and then apply supervised fine-tuning or preference
optimization based on these outputs. Extensive experiments on several leading
LLMs demonstrate the effectiveness of \ours, with an absolute improvement of
4.2 points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior
performance compared to prior approaches that depend on data produced by human
experts or advanced models. We anticipate that this work will open new avenues
for self-improvement techniques in long-context scenarios, which are essential
for the continual advancement of LLMs.
673593ee8d2bfc6159c2f994
Siheng Li
extracted_confirmed
2024-11-14T06:09:11.650Z
66d45a8de5837f38ce3b73f7
https://cdn-avatars.huggingface.co/v1/production/uploads/66d45a8de5837f38ce3b73f7/3omslNRb8wV_c1xbrCmQC.jpeg
SihengLi
user
Siheng99
673593ee8d2bfc6159c2f995
Cheng Yang
claimed_verified
2024-11-17T16:18:14.444Z
64b75db417570fdff9b43fd9
https://cdn-avatars.huggingface.co/v1/production/uploads/64b75db417570fdff9b43fd9/-NcNL_QzuiAAvsBIQG3xo.png
Cheng YANG
user
Cheng-YANG
673593ee8d2bfc6159c2f996
Zesen Cheng
claimed_verified
2024-11-15T09:21:54.860Z
65b2529285b6c21448a10d65
/avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
Zesen Cheng
user
ClownRat
673593ee8d2bfc6159c2f997
Lemao Liu
admin_assigned
2024-11-15T14:09:49.517Z
64462d77efa0723f3946356b
/avatars/f9757030d82c69aef933309e0c83ccd0.svg
Lemao Liu
user
lemaoliu
673593ee8d2bfc6159c2f998
Mo Yu
claimed_verified
2025-02-18T09:36:25.917Z
67af92045a86287292026808
/avatars/8bad9272fe73ba04e077b5484837c8d3.svg
Mo
user
BishopGorov
673593ee8d2bfc6159c2f999
Yujiu Yang
admin_assigned
2024-11-15T14:10:07.290Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
673593ee8d2bfc6159c2f99a
Wai Lam
Large Language Models Can Self-Improve in Long-context Reasoning
Large language models (LLMs) have achieved substantial progress in processing
long contexts but still struggle with long-context reasoning. Existing
approaches typically involve fine-tuning LLMs with synthetic data, which
depends on annotations from human experts or advanced models like GPT-4, thus
restricting further advancements. To address this issue, we investigate the
potential for LLMs to self-improve in long-context reasoning and propose \ours,
an approach specifically designed for this purpose. This approach is
straightforward: we sample multiple outputs for each question, score them with
Minimum Bayes Risk, and then apply supervised fine-tuning or preference
optimization based on these outputs. Extensive experiments on several leading
LLMs demonstrate the effectiveness of \ours, with an absolute improvement of
4.2 points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior
performance compared to prior approaches that depend on data produced by human
experts or advanced models. We anticipate that this work will open new avenues
for self-improvement techniques in long-context scenarios, which are essential
for the continual advancement of LLMs.
A novel method enables LLMs to self-improve in long-context reasoning through sampling, scoring with Minimum Bayes Risk, and supervised fine-tuning, outperforming expert-annotated data approaches.
LLMs long-context reasoning fine-tuning synthetic data GPT-4 Minimum Bayes Risk supervised fine-tuning preference optimization Llama-3.1-8B-Instruct