CodecLM: Aligning Language Models with Tailored Synthetic Data
Instruction tuning has emerged as the key in aligning large language models
(LLMs) with specific task instructions, thereby mitigating the discrepancy
between the next-token prediction objective and users' actual goals. To reduce
the labor and time cost to collect or annotate data by humans, researchers
start to explore the use of LLMs to generate instruction-aligned synthetic
data. Recent works focus on generating diverse instructions and applying LLM to
increase instruction complexity, often neglecting downstream use cases. It
remains unclear how to tailor high-quality data to elicit better
instruction-following abilities in different target instruction distributions
and LLMs. To this end, we introduce CodecLM, a general framework for adaptively
generating high-quality synthetic data for LLM alignment with different
downstream instruction distributions and LLMs. Drawing on the Encode-Decode
principles, we use LLMs as codecs to guide the data generation process. We
first encode seed instructions into metadata, which are concise keywords
generated on-the-fly to capture the target instruction distribution, and then
decode metadata to create tailored instructions. We also introduce Self-Rubrics
and Contrastive Filtering during decoding to tailor data-efficient samples.
Extensive experiments on four open-domain instruction following benchmarks
validate the effectiveness of CodecLM over the current state-of-the-arts.
6615f4b861fb2574d52a37ed
Zifeng Wang
claimed_verified
2024-04-12T19:56:05.811Z
64da7b3bbb090cef5559fa31
/avatars/4eb059260f06837192c98927254cd89e.svg
Zifeng Wang
user
zifengw
6615f4b861fb2574d52a37ee
Chun-Liang Li
6615f4b861fb2574d52a37ef
Vincent Perot
claimed_verified
2024-04-11T05:52:30.935Z
64e3f501bbee0c24b95e20c2
/avatars/bae7b5297e779d91da21ff50389dd90a.svg
Vincent
user
vperot
6615f4b861fb2574d52a37f0
Long T. Le
6615f4b861fb2574d52a37f1
Jin Miao
6615f4b861fb2574d52a37f2
Zizhao Zhang
6615f4b861fb2574d52a37f3
Chen-Yu Lee
claimed_verified
2024-07-24T07:29:12.475Z
64d1fe784dfd5df7076bfd1a
/avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
Chen-Yu Lee
user
chenyulee
6615f4b861fb2574d52a37f4
Tomas Pfister
CodecLM: Aligning Language Models with Tailored Synthetic Data
Instruction tuning has emerged as the key in aligning large language models
(LLMs) with specific task instructions, thereby mitigating the discrepancy
between the next-token prediction objective and users' actual goals. To reduce
the labor and time cost to collect or annotate data by humans, researchers
start to explore the use of LLMs to generate instruction-aligned synthetic
data. Recent works focus on generating diverse instructions and applying LLM to
increase instruction complexity, often neglecting downstream use cases. It
remains unclear how to tailor high-quality data to elicit better
instruction-following abilities in different target instruction distributions
and LLMs. To this end, we introduce CodecLM, a general framework for adaptively
generating high-quality synthetic data for LLM alignment with different
downstream instruction distributions and LLMs. Drawing on the Encode-Decode
principles, we use LLMs as codecs to guide the data generation process. We
first encode seed instructions into metadata, which are concise keywords
generated on-the-fly to capture the target instruction distribution, and then
decode metadata to create tailored instructions. We also introduce Self-Rubrics
and Contrastive Filtering during decoding to tailor data-efficient samples.
Extensive experiments on four open-domain instruction following benchmarks
validate the effectiveness of CodecLM over the current state-of-the-arts.
CodecLM is a framework that utilizes LLMs to generate high-quality, aligned synthetic data for instruction-following tasks, improving adaptability and performance across various LLMs and instruction distributions.
instruction tuning large language models (LLMs) synthetic data Encode-Decode principles codecs metadata self-rubrics contrastive filtering instruction-following benchmarks