IOPO: Empowering LLMs with Complex Instruction Following via
  Input-Output Preference Optimization
In the realm of large language models (LLMs), the ability of models to
accurately follow instructions is paramount as more agents and applications
leverage LLMs for construction, where the complexity of instructions are
rapidly increasing. However, on the one hand, there is only a certain amount of
complex instruction evaluation data; on the other hand, there are no dedicated
algorithms to improve the ability to follow complex instructions. To this end,
this paper introduces TRACE, a benchmark for improving and evaluating the
complex instructionfollowing ability, which consists of 120K training data and
1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference
Optimization) alignment method which takes both input and output preference
pairs into consideration, where LLMs not only rapidly align with response
preferences but also meticulously explore the instruction preferences.
Extensive experiments on both in-domain and outof-domain datasets confirm the
effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and
6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.
6732d357faded36d0ebfb787
Xinghua Zhang
6732d357faded36d0ebfb788
Haiyang Yu
6732d357faded36d0ebfb789
Cheng Fu
6732d357faded36d0ebfb78a
Fei Huang
6732d357faded36d0ebfb78b
Yongbin Li
IOPO: Empowering LLMs with Complex Instruction Following via
  Input-Output Preference Optimization
In the realm of large language models (LLMs), the ability of models to
accurately follow instructions is paramount as more agents and applications
leverage LLMs for construction, where the complexity of instructions are
rapidly increasing. However, on the one hand, there is only a certain amount of
complex instruction evaluation data; on the other hand, there are no dedicated
algorithms to improve the ability to follow complex instructions. To this end,
this paper introduces TRACE, a benchmark for improving and evaluating the
complex instructionfollowing ability, which consists of 120K training data and
1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference
Optimization) alignment method which takes both input and output preference
pairs into consideration, where LLMs not only rapidly align with response
preferences but also meticulously explore the instruction preferences.
Extensive experiments on both in-domain and outof-domain datasets confirm the
effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and
6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.
TRACE is a benchmark for improving complex instruction-following in LLMs, using IOPO to enhance input-output alignment, demonstrating superior performance over SFT and DPO.
large language models (LLMs) TRACE benchmark complex instruction-following IOPO (Input-Output Preference Optimization) in-domain out-of-domain SFT DPO