CodeChain: Towards Modular Code Generation Through Chain of
  Self-revisions with Representative Sub-modules
Large Language Models (LLMs) have already become quite proficient at solving
simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
solving more complex and competitive programming tasks is still quite
challenging for these models - possibly due to their tendency to generate
solutions as monolithic code blocks instead of decomposing them into logical
sub-tasks and sub-modules. On the other hand, experienced programmers
instinctively write modularized code with abstraction for solving complex
tasks, often reusing previously developed modules. To address this gap, we
propose CodeChain, a novel framework for inference that elicits modularized
code generation through a chain of self-revisions, each being guided by some
representative sub-modules generated in previous iterations. Concretely,
CodeChain first instructs the LLM to generate modularized codes through
chain-of-thought prompting. Then it applies a chain of self-revisions by
iterating the two steps: 1) extracting and clustering the generated sub-modules
and selecting the cluster representatives as the more generic and re-usable
implementations, and 2) augmenting the original chain-of-thought prompt with
these selected module-implementations and instructing the LLM to re-generate
new modularized solutions. We find that by naturally encouraging the LLM to
reuse the previously developed and verified sub-modules, CodeChain can
significantly boost both modularity as well as correctness of the generated
solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
CodeContests. It is shown to be effective on both OpenAI LLMs as well as
open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
studies with different methods of prompting, number of clusters, model sizes,
program qualities, etc., to provide useful insights that underpin CodeChain's
success.
652cc1825aec376f553d4dfe
Hung Le
claimed_verified
2023-10-19T13:37:11.000Z
63eeee92be5d5900b85367bb
/avatars/ef10dd6e1ca85861996034797cafe5b8.svg
Hung Le
user
hle2023
652cc1825aec376f553d4dff
Hailin Chen
admin_assigned
2023-10-16T08:25:05.749Z
646c440036505117e22f5b76
/avatars/a833947c9d452d673d2598eea7363a73.svg
Hailin Chen
user
hailche
652cc1825aec376f553d4e00
Amrita Saha
admin_assigned
2023-10-16T08:25:14.238Z
6461c2905dba83471db3be53
/avatars/6e36cf86201d590ac729a75d4a439cde.svg
Amrita Saha
user
amritasaha87
652cc1825aec376f553d4e01
Akash Gokul
admin_assigned
2023-10-16T08:25:21.435Z
63093856b105f8675bd1c783
/avatars/505a7b99893330a36a136d13fa99e05e.svg
Akash Gokul
user
akashgokul
652cc1825aec376f553d4e02
Doyen Sahoo
652cc1825aec376f553d4e03
Shafiq Joty
CodeChain: Towards Modular Code Generation Through Chain of
  Self-revisions with Representative Sub-modules
Large Language Models (LLMs) have already become quite proficient at solving
simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
solving more complex and competitive programming tasks is still quite
challenging for these models - possibly due to their tendency to generate
solutions as monolithic code blocks instead of decomposing them into logical
sub-tasks and sub-modules. On the other hand, experienced programmers
instinctively write modularized code with abstraction for solving complex
tasks, often reusing previously developed modules. To address this gap, we
propose CodeChain, a novel framework for inference that elicits modularized
code generation through a chain of self-revisions, each being guided by some
representative sub-modules generated in previous iterations. Concretely,
CodeChain first instructs the LLM to generate modularized codes through
chain-of-thought prompting. Then it applies a chain of self-revisions by
iterating the two steps: 1) extracting and clustering the generated sub-modules
and selecting the cluster representatives as the more generic and re-usable
implementations, and 2) augmenting the original chain-of-thought prompt with
these selected module-implementations and instructing the LLM to re-generate
new modularized solutions. We find that by naturally encouraging the LLM to
reuse the previously developed and verified sub-modules, CodeChain can
significantly boost both modularity as well as correctness of the generated
solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
CodeContests. It is shown to be effective on both OpenAI LLMs as well as
open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
studies with different methods of prompting, number of clusters, model sizes,
program qualities, etc., to provide useful insights that underpin CodeChain's
success.
CodeChain improves the modularity and correctness of solutions generated by Large Language Models (LLMs) through a chain of self-revisions and sub-module reuse, achieving significant performance gains on competitive programming tasks.
Large Language Models LLMs HumanEval MBPP modularized code chain-of-thought prompting self-revisions sub-modules abstraction sub-module reuse chain-of-thought CodeChain APPS CodeContests OpenAI WizardCoder ablation studies