Language Models as Compilers: Simulating Pseudocode Execution Improves
  Algorithmic Reasoning in Language Models
Algorithmic reasoning refers to the ability to understand the complex
patterns behind the problem and decompose them into a sequence of reasoning
steps towards the solution. Such nature of algorithmic reasoning makes it a
challenge for large language models (LLMs), even though they have demonstrated
promising performance in other reasoning tasks. Within this context, some
recent studies use programming languages (e.g., Python) to express the
necessary logic for solving a given instance/question (e.g.,
Program-of-Thought) as inspired by their strict and precise syntaxes. However,
it is non-trivial to write an executable code that expresses the correct logic
on the fly within a single inference call. Also, the code generated
specifically for an instance cannot be reused for others, even if they are from
the same task and might require identical logic to solve. This paper presents
Think-and-Execute, a novel framework that decomposes the reasoning process of
language models into two steps. (1) In Think, we discover a task-level logic
that is shared across all instances for solving a given task and then express
the logic with pseudocode; (2) In Execute, we further tailor the generated
pseudocode to each instance and simulate the execution of the code. With
extensive experiments on seven algorithmic reasoning tasks, we demonstrate the
effectiveness of Think-and-Execute. Our approach better improves LMs' reasoning
compared to several strong baselines performing instance-specific reasoning
(e.g., CoT and PoT), suggesting the helpfulness of discovering task-level
logic. Also, we show that compared to natural language, pseudocode can better
guide the reasoning of LMs, even though they are trained to follow natural
language instructions.
660e2ce3e23148df7c927a69
Hyungjoo Chae
claimed_verified
2024-04-04T07:56:48.638Z
64c8f4cec547ed5243ebd0a8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8f4cec547ed5243ebd0a8/MiOH5YbMg8Gh9KYlQsLmX.jpeg
Hyungjoo Chae
user
hyungjoochae
660e2ce3e23148df7c927a6a
Yeonghyeon Kim
claimed_verified
2024-04-15T06:45:53.955Z
65951e629b53611f666c7973
/avatars/a395a480ef35c84bd2df204f10a6e873.svg
Yeong-Hyeon Kim
user
younghkim1
660e2ce3e23148df7c927a6b
Seungone Kim
claimed_verified
2024-04-05T10:06:14.039Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
660e2ce3e23148df7c927a6c
Kai Tzu-iunn Ong
claimed_verified
2024-04-05T10:05:55.404Z
640ec2fd2f9c7b364d182735
https://cdn-avatars.huggingface.co/v1/production/uploads/640ec2fd2f9c7b364d182735/G5WxQXYEOaVL39rAoOI3Y.jpeg
kai tzu-iunn ong
user
ktio
660e2ce3e23148df7c927a6d
Beong-woo Kwak
660e2ce3e23148df7c927a6e
Moohyeon Kim
claimed_verified
2024-05-07T10:24:12.229Z
659513f0f7291078f9ae8c34
/avatars/2c58fc0630c391a0ae86851fef77898b.svg
Moohyeon Kim
user
moojuice8
660e2ce3e23148df7c927a6f
Seonghwan Kim
660e2ce3e23148df7c927a70
Taeyoon Kwon
claimed_verified
2025-05-27T07:59:32.611Z
636b529ef796304dd67d139c
/avatars/7a64d5095fcb1da558b52ad48177ad76.svg
Taeyoon Kwon
user
Connoriginal
660e2ce3e23148df7c927a71
Jiwan Chung
claimed_verified
2025-06-03T09:18:00.989Z
60d74d1affe0328e0167dc5f
/avatars/9b1a2df9402e9c26e1eb7c818af9bae0.svg
Jiwan Chung
user
jiwan-chung
660e2ce3e23148df7c927a72
Youngjae Yu
660e2ce3e23148df7c927a73
Jinyoung Yeo
Language Models as Compilers: Simulating Pseudocode Execution Improves
  Algorithmic Reasoning in Language Models
Algorithmic reasoning refers to the ability to understand the complex
patterns behind the problem and decompose them into a sequence of reasoning
steps towards the solution. Such nature of algorithmic reasoning makes it a
challenge for large language models (LLMs), even though they have demonstrated
promising performance in other reasoning tasks. Within this context, some
recent studies use programming languages (e.g., Python) to express the
necessary logic for solving a given instance/question (e.g.,
Program-of-Thought) as inspired by their strict and precise syntaxes. However,
it is non-trivial to write an executable code that expresses the correct logic
on the fly within a single inference call. Also, the code generated
specifically for an instance cannot be reused for others, even if they are from
the same task and might require identical logic to solve. This paper presents
Think-and-Execute, a novel framework that decomposes the reasoning process of
language models into two steps. (1) In Think, we discover a task-level logic
that is shared across all instances for solving a given task and then express
the logic with pseudocode; (2) In Execute, we further tailor the generated
pseudocode to each instance and simulate the execution of the code. With
extensive experiments on seven algorithmic reasoning tasks, we demonstrate the
effectiveness of Think-and-Execute. Our approach better improves LMs' reasoning
compared to several strong baselines performing instance-specific reasoning
(e.g., CoT and PoT), suggesting the helpfulness of discovering task-level
logic. Also, we show that compared to natural language, pseudocode can better
guide the reasoning of LMs, even though they are trained to follow natural
language instructions.
Think-and-Execute is a novel framework that enhances algorithmic reasoning in large language models by decomposing the reasoning process into task-level logic expressed as pseudocode and instance-specific execution.
algorithmic reasoning large language models LLMs programming languages Python Program-of-Thought pseudocode Think-and-Execute