Branch-Solve-Merge Improves Large Language Model Evaluation and
  Generation
Large Language Models (LLMs) are frequently used for multi-faceted language
generation and evaluation tasks that involve satisfying intricate user
constraints or taking into account multiple aspects and criteria. However,
their performance can fall short, due to the model's lack of coherence and
inability to plan and decompose the problem. We propose Branch-Solve-Merge
(BSM), a Large Language Model program (Schlag et al., 2023) for tackling such
challenging natural language tasks. It consists of branch, solve, and merge
modules that are parameterized with specific prompts to the base LLM. These
three modules plan a decomposition of the task into multiple parallel
sub-tasks, independently solve them, and fuse the solutions to the sub-tasks.
We apply our method to the tasks of LLM response evaluation and constrained
text generation and evaluate its effectiveness with multiple LLMs, including
Vicuna, LLaMA-2-chat, and GPT-4. BSM improves the evaluation correctness and
consistency for each LLM by enhancing human-LLM agreement by up to 26%,
reducing length and pairwise position biases by up to 50%, and allowing
LLaMA-2-chat to match or outperform GPT-4 on most domains. On the constraint
story generation task, BSM improves the coherence of the stories while also
improving constraint satisfaction by 12%.
65373db1d94ca568059aa723
Swarnadeep Saha
admin_assigned
2023-10-24T08:59:18.382Z
64b6feee17681d64b19b112b
/avatars/afe708392d16af5b55d3bc4b42a585e8.svg
Swarnadeep Saha
user
swarnaNLP
65373db1d94ca568059aa724
Omer Levy
admin_assigned
2023-10-24T08:59:24.076Z
610b71d7f5dd7bcefedaff55
/avatars/b489c021e906010e661313da0b4aafab.svg
Omer Levy
user
omerlevy
65373db1d94ca568059aa725
Asli Celikyilmaz
65373db1d94ca568059aa726
Mohit Bansal
65373db1d94ca568059aa727
Jason Weston
admin_assigned
2023-10-24T09:00:09.336Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
65373db1d94ca568059aa728
Xian Li
Branch-Solve-Merge Improves Large Language Model Evaluation and
  Generation
Large Language Models (LLMs) are frequently used for multi-faceted language
generation and evaluation tasks that involve satisfying intricate user
constraints or taking into account multiple aspects and criteria. However,
their performance can fall short, due to the model's lack of coherence and
inability to plan and decompose the problem. We propose Branch-Solve-Merge
(BSM), a Large Language Model program (Schlag et al., 2023) for tackling such
challenging natural language tasks. It consists of branch, solve, and merge
modules that are parameterized with specific prompts to the base LLM. These
three modules plan a decomposition of the task into multiple parallel
sub-tasks, independently solve them, and fuse the solutions to the sub-tasks.
We apply our method to the tasks of LLM response evaluation and constrained
text generation and evaluate its effectiveness with multiple LLMs, including
Vicuna, LLaMA-2-chat, and GPT-4. BSM improves the evaluation correctness and
consistency for each LLM by enhancing human-LLM agreement by up to 26%,
reducing length and pairwise position biases by up to 50%, and allowing
LLaMA-2-chat to match or outperform GPT-4 on most domains. On the constraint
story generation task, BSM improves the coherence of the stories while also
improving constraint satisfaction by 12%.
The Branch-Solve-Merge (BSM) method enhances Large Language Models for complex tasks by decomposing them into sub-tasks, solving each independently, and then merging the results, improving evaluation and constrained text generation.
Large Language Models LLMs branch solve merge BSM LLM response evaluation constrained text generation Vicuna LLaMA-2-chat GPT-4 human-LLM agreement coherence constraint satisfaction constraint story generation