WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with
  Refined Data Generation
Recent work demonstrates that, after being fine-tuned on a high-quality
instruction dataset, the resulting model can obtain impressive capabilities to
address a wide range of tasks. However, existing methods for instruction data
generation often produce duplicate data and are not controllable enough on data
quality. In this paper, we extend the generalization of instruction tuning by
classifying the instruction data to 4 code-related tasks and propose a
LLM-based Generator-Discriminator data process framework to generate diverse,
high-quality instruction data from open source code. Hence, we introduce
CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal
code-related tasks,which is aimed at augmenting the effectiveness of
instruction tuning and improving the generalization ability of fine-tuned
model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with
Widespread And Versatile Enhanced instruction tuning. This model is
specifically designed for enhancing instruction tuning of Code Language Models
(LLMs). Our experiments demonstrate that Wavecoder models outperform other
open-source models in terms of generalization ability across different
code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder
exhibits high efficiency in previous code generation tasks. This paper thus
offers a significant contribution to the field of instruction data generation
and fine-tuning models, providing new insights and tools for enhancing
performance in code-related tasks.
658a4687eac1c5dac0efee86
Zhaojian Yu
claimed_verified
2023-12-26T10:37:01.996Z
646f3443c261dc413383b8a4
https://cdn-avatars.huggingface.co/v1/production/uploads/646f3443c261dc413383b8a4/7R2XX3VqQ6DmgQoMU7Un4.jpeg
Zhaojian Yu
user
zjy2001
658a4687eac1c5dac0efee87
Xin Zhang
658a4687eac1c5dac0efee88
Ning Shang
658a4687eac1c5dac0efee89
Yangyu Huang
claimed_verified
2025-01-13T13:03:19.388Z
64c66647725ffa04b2fd6c94
/avatars/620f63f27fa1e90423b0dc22aa8e5809.svg
yangyu huang
user
yangyu90
658a4687eac1c5dac0efee8a
Can Xu
658a4687eac1c5dac0efee8b
Yishujie Zhao
658a4687eac1c5dac0efee8c
Wenxiang Hu
claimed_verified
2024-01-03T12:36:18.126Z
6327cafc46ffcfc27cfa4c9a
/avatars/4daea27b432ff428c41d590669f7330c.svg
Wenxiang Hu
user
wenxcs
658a4687eac1c5dac0efee8d
Qiufeng Yin
WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with
  Refined Data Generation
Recent work demonstrates that, after being fine-tuned on a high-quality
instruction dataset, the resulting model can obtain impressive capabilities to
address a wide range of tasks. However, existing methods for instruction data
generation often produce duplicate data and are not controllable enough on data
quality. In this paper, we extend the generalization of instruction tuning by
classifying the instruction data to 4 code-related tasks and propose a
LLM-based Generator-Discriminator data process framework to generate diverse,
high-quality instruction data from open source code. Hence, we introduce
CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal
code-related tasks,which is aimed at augmenting the effectiveness of
instruction tuning and improving the generalization ability of fine-tuned
model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with
Widespread And Versatile Enhanced instruction tuning. This model is
specifically designed for enhancing instruction tuning of Code Language Models
(LLMs). Our experiments demonstrate that Wavecoder models outperform other
open-source models in terms of generalization ability across different
code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder
exhibits high efficiency in previous code generation tasks. This paper thus
offers a significant contribution to the field of instruction data generation
and fine-tuning models, providing new insights and tools for enhancing
performance in code-related tasks.
A Generator-Discriminator framework using LLMs generates high-quality instruction data, enhancing the fine-tuning of code-related tasks and model generalization.
instruction tuning code-related tasks Generator-Discriminator framework LLM-based CodeOcean instruction data fine-tuned Code LLM WaveCoder generalization ability