Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
Code provides a general syntactic structure to build complex programs and
perform precise computations when paired with a code interpreter -- we
hypothesize that language models (LMs) can leverage code-writing to improve
Chain of Thought reasoning not only for logic and arithmetic tasks, but also
for linguistic ones (and in particular, those that are a mix of both). For
example, consider prompting an LM to write code that counts the number of times
it detects sarcasm in an essay: the LM may struggle to write an implementation
for "detect_sarcasm(string)" that can be executed by the interpreter (handling
the edge cases would be insurmountable). However, LMs may still produce a valid
solution if they are used not only to write the code, but also to selectively
"emulate" the interpreter by generating the expected output of
"detect_sarcasm(string)" and other lines of code (e.g., that the interpreter
could not compile). In this work, we propose Chain of Code (CoT), a simple yet
surprisingly effective extension that improves LM code-driven reasoning. The
key idea is to encourage LMs to format linguistic sub-tasks in a program as
flexible pseudocode that the compiler can explicitly catch undefined behaviors
and hand off to simulate with an LM (as an "LMulator"). Experiments demonstrate
that Chain of Code outperforms Chain of Thought and other baselines across a
variety of benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of
12% over Chain of Thought. CoT scales well with large and small models alike,
and broadens the scope of reasoning questions that LMs can correctly answer by
"thinking in code". Project webpage: https://chain-of-code.github.io/.
65727dbec2913e69b4c5c215
Chengshu Li
extracted_confirmed
2024-05-22T00:20:40.845Z
664d3a3079f09472c1193e06
/avatars/3cf4591f36230137bb059e7f2de8bbc3.svg
ChengshuLi
user
chengshu
65727dbec2913e69b4c5c216
Jacky Liang
65727dbec2913e69b4c5c217
Andy Zeng
admin_assigned
2023-12-08T10:59:43.557Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
65727dbec2913e69b4c5c218
Xinyun Chen
admin_assigned
2023-12-08T11:02:55.074Z
64d0268001931c60161e026a
/avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
Xinyun Chen
user
xinyunchen
65727dbec2913e69b4c5c219
Karol Hausman
admin_assigned
2023-12-08T11:03:03.055Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
65727dbec2913e69b4c5c21a
Dorsa Sadigh
65727dbec2913e69b4c5c21b
Sergey Levine
65727dbec2913e69b4c5c21c
Li Fei-Fei
65727dbec2913e69b4c5c21d
Fei Xia
claimed_verified
2023-12-08T08:00:20.825Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
65727dbec2913e69b4c5c21e
Brian Ichter
admin_assigned
2023-12-08T11:03:29.078Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
Code provides a general syntactic structure to build complex programs and
perform precise computations when paired with a code interpreter -- we
hypothesize that language models (LMs) can leverage code-writing to improve
Chain of Thought reasoning not only for logic and arithmetic tasks, but also
for linguistic ones (and in particular, those that are a mix of both). For
example, consider prompting an LM to write code that counts the number of times
it detects sarcasm in an essay: the LM may struggle to write an implementation
for "detect_sarcasm(string)" that can be executed by the interpreter (handling
the edge cases would be insurmountable). However, LMs may still produce a valid
solution if they are used not only to write the code, but also to selectively
"emulate" the interpreter by generating the expected output of
"detect_sarcasm(string)" and other lines of code (e.g., that the interpreter
could not compile). In this work, we propose Chain of Code (CoT), a simple yet
surprisingly effective extension that improves LM code-driven reasoning. The
key idea is to encourage LMs to format linguistic sub-tasks in a program as
flexible pseudocode that the compiler can explicitly catch undefined behaviors
and hand off to simulate with an LM (as an "LMulator"). Experiments demonstrate
that Chain of Code outperforms Chain of Thought and other baselines across a
variety of benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of
12% over Chain of Thought. CoT scales well with large and small models alike,
and broadens the scope of reasoning questions that LMs can correctly answer by
"thinking in code". Project webpage: https://chain-of-code.github.io/.
language models (LMs) code-writing Chain of Thought reasoning synthesis computational requirements interpreter code "detect_sarcasm(string)" Con insurmountable Chain of Code (CoT) pseudocode undefined behaviors LMulator BIG-Bench Hard