From Real to Synthetic: Synthesizing Millions of Diversified and
  Complicated User Instructions with Attributed Grounding
The pursuit of diverse, complex, and large-scale instruction data is crucial
for automatically aligning large language models (LLMs). While there are
methods capable of generating synthetic instructions at scale, they either
suffer from limited grounding sources, leading to a narrow distribution, or
rely on trivial extensions that fail to produce meaningful trajectories in
terms of complexity. In contrast, instructions that benefit efficient alignment
are typically crafted with cognitive insights and grounded in real-world use
cases. In this paper, we synthesize such instructions using attributed
grounding, which involves 1) a top-down attribution process that grounds a
selective set of real instructions to situated users, and 2) a bottom-up
synthesis process that leverages web documents to first generate a situation,
then a meaningful instruction. This framework allows us to harvest diverse and
complex instructions at scale, utilizing the vast range of web documents.
Specifically, we construct a dataset of 1 million instructions, called
SynthQuestions, and demonstrate that models trained on it achieve leading
performance on several common benchmarks, with improvements that continually
scale with more web corpora. Data, models and codes will be available at
https://github.com/Ignoramus0817/SynthQuestions.
684169b041d567923aa6c5be
Chiwei Zhu
claimed_verified
2025-06-05T09:59:41.997Z
663b22a80966eef8686aadaf
https://cdn-avatars.huggingface.co/v1/production/uploads/663b22a80966eef8686aadaf/iBzyQTyGZKf33RPVIFh9a.jpeg
Chiwei Zhu
user
IgnoraZ
684169b041d567923aa6c5bf
Benfeng Xu
684169b041d567923aa6c5c0
Xiaorui Wang
684169b041d567923aa6c5c1
Zhendong Mao
From Real to Synthetic: Synthesizing Millions of Diversified and
  Complicated User Instructions with Attributed Grounding
The pursuit of diverse, complex, and large-scale instruction data is crucial
for automatically aligning large language models (LLMs). While there are
methods capable of generating synthetic instructions at scale, they either
suffer from limited grounding sources, leading to a narrow distribution, or
rely on trivial extensions that fail to produce meaningful trajectories in
terms of complexity. In contrast, instructions that benefit efficient alignment
are typically crafted with cognitive insights and grounded in real-world use
cases. In this paper, we synthesize such instructions using attributed
grounding, which involves 1) a top-down attribution process that grounds a
selective set of real instructions to situated users, and 2) a bottom-up
synthesis process that leverages web documents to first generate a situation,
then a meaningful instruction. This framework allows us to harvest diverse and
complex instructions at scale, utilizing the vast range of web documents.
Specifically, we construct a dataset of 1 million instructions, called
SynthQuestions, and demonstrate that models trained on it achieve leading
performance on several common benchmarks, with improvements that continually
scale with more web corpora. Data, models and codes will be available at
https://github.com/Ignoramus0817/SynthQuestions.
The paper presents a method for generating diverse and complex instruction data for large language models using attributed grounding, achieving top performance on benchmarks with a large synthesized dataset.
acknowledged grounding top-down attribution process bottom-up synthesis process web documents large language models SynthQuestions