The Impact of Reasoning Step Length on Large Language Models
Chain of Thought (CoT) is significant in improving the reasoning abilities of
large language models (LLMs). However, the correlation between the
effectiveness of CoT and the length of reasoning steps in prompts remains
largely unknown. To shed light on this, we have conducted several empirical
experiments to explore the relations. Specifically, we design experiments that
expand and compress the rationale reasoning steps within CoT demonstrations,
while keeping all other factors constant. We have the following key findings.
First, the results indicate that lengthening the reasoning steps in prompts,
even without adding new information into the prompt, considerably enhances
LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
the reasoning steps, even while preserving the key information, significantly
diminishes the reasoning abilities of models. This finding highlights the
importance of the number of steps in CoT prompts and provides practical
guidance to make better use of LLMs' potential in complex problem-solving
scenarios. Second, we also investigated the relationship between the
performance of CoT and the rationales used in demonstrations. Surprisingly, the
result shows that even incorrect rationales can yield favorable outcomes if
they maintain the requisite length of inference. Third, we observed that the
advantages of increasing reasoning steps are task-dependent: simpler tasks
require fewer steps, whereas complex tasks gain significantly from longer
inference sequences.
659f5b037edb2a072edfaca6
Mingyu Jin
admin_assigned
2024-01-11T10:54:45.675Z
653b47abf44bc2eacef5c05f
/avatars/aa4e3a9dd2e4c6c474790f169b70d52e.svg
Mingyu.Jin
user
Mingyu111
659f5b037edb2a072edfaca7
Qinkai Yu
659f5b037edb2a072edfaca8
Dong shu
659f5b037edb2a072edfaca9
Haiyan Zhao
659f5b037edb2a072edfacaa
Wenyue Hua
admin_assigned
2024-01-11T10:55:32.448Z
639a25aba2b0b1c9d85a51e8
https://cdn-avatars.huggingface.co/v1/production/uploads/639a25aba2b0b1c9d85a51e8/pphz-MK62hPNbBkMHAkeR.jpeg
Wenyue Hua
user
wenyueH
659f5b037edb2a072edfacab
Yanda Meng
admin_assigned
2024-01-11T10:55:39.764Z
64a7e092948958fd727b8204
/avatars/1b52669fd3d01947a8035462e7063aea.svg
yanda meng
user
bigmaxmax
659f5b037edb2a072edfacac
Yongfeng Zhang
admin_assigned
2024-01-11T10:55:46.463Z
63b0deb7f2eb87a4d695055c
/avatars/85a87fa82f7579019a0c6212f03b6480.svg
Yongfeng Zhang
user
evison
659f5b037edb2a072edfacad
Mengnan Du
The Impact of Reasoning Step Length on Large Language Models
Chain of Thought (CoT) is significant in improving the reasoning abilities of
large language models (LLMs). However, the correlation between the
effectiveness of CoT and the length of reasoning steps in prompts remains
largely unknown. To shed light on this, we have conducted several empirical
experiments to explore the relations. Specifically, we design experiments that
expand and compress the rationale reasoning steps within CoT demonstrations,
while keeping all other factors constant. We have the following key findings.
First, the results indicate that lengthening the reasoning steps in prompts,
even without adding new information into the prompt, considerably enhances
LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
the reasoning steps, even while preserving the key information, significantly
diminishes the reasoning abilities of models. This finding highlights the
importance of the number of steps in CoT prompts and provides practical
guidance to make better use of LLMs' potential in complex problem-solving
scenarios. Second, we also investigated the relationship between the
performance of CoT and the rationales used in demonstrations. Surprisingly, the
result shows that even incorrect rationales can yield favorable outcomes if
they maintain the requisite length of inference. Third, we observed that the
advantages of increasing reasoning steps are task-dependent: simpler tasks
require fewer steps, whereas complex tasks gain significantly from longer
inference sequences.
Increasing the length of reasoning steps in Chain of Thought prompts improves large language models' reasoning abilities, even with incorrect rationales, and this benefit is task-dependent.
Chain of Thought large language models reasoning abilities rationale reasoning steps empirical experiments inference sequences