Selecting Influential Samples for Long Context Alignment via Homologous
  Models' Guidance and Contextual Awareness Measurement
The expansion of large language models to effectively handle instructions
with extremely long contexts has yet to be fully investigated. The primary
obstacle lies in constructing a high-quality long instruction-following dataset
devised for long context alignment. Existing studies have attempted to scale up
the available data volume by synthesizing long instruction-following samples.
However, indiscriminately increasing the quantity of data without a
well-defined strategy for ensuring data quality may introduce low-quality
samples and restrict the final performance. To bridge this gap, we aim to
address the unique challenge of long-context alignment, i.e., modeling the
long-range dependencies for handling instructions and lengthy input contexts.
We propose GATEAU, a novel framework designed to identify the influential and
high-quality samples enriched with long-range dependency relations by utilizing
crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
(CAM). Specifically, HMG attempts to measure the difficulty of generating
corresponding responses due to the long-range dependencies, using the
perplexity scores of the response from two homologous models with different
context windows. Also, the role of CAM is to measure the difficulty of
understanding the long input contexts due to long-range dependencies by
evaluating whether the model's attention is focused on important segments.
Built upon both proposed methods, we select the most challenging samples as the
influential data to effectively frame the long-range dependencies, thereby
achieving better performance of LLMs. Comprehensive experiments indicate that
GATEAU effectively identifies samples enriched with long-range dependency
relations and the model trained on these selected samples exhibits better
instruction-following and long-context understanding capabilities.
671742ab0c095b8e5d099551
Shuzheng Si
claimed_verified
2024-10-22T15:02:35.806Z
637c99bbfe115289cfedfb44
https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
ssz
user
ssz1111
671742ab0c095b8e5d099552
Haozhe Zhao
claimed_verified
2025-05-26T08:45:25.748Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
671742ab0c095b8e5d099553
Gang Chen
671742ab0c095b8e5d099554
Yunshui Li
admin_assigned
2024-10-22T13:29:52.787Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
671742ab0c095b8e5d099555
Kangyang Luo
admin_assigned
2024-10-22T13:29:58.585Z
66ba08d4bafd993a5428e051
/avatars/2bd7779582ee3ae2ff9688e1627d745e.svg
Kangyang Luo
user
lKangyang
671742ab0c095b8e5d099556
Chuancheng Lv
671742ab0c095b8e5d099557
Kaikai An
claimed_verified
2024-10-23T07:34:44.942Z
66c89152d33e34fbc29497d7
/avatars/bbddabf6532393951c4759e5915a065b.svg
KaikaiAn
user
kkk-an
671742ab0c095b8e5d099558
Fanchao Qi
admin_assigned
2024-10-22T13:29:21.938Z
65623af21a0c459616cc4b03
/avatars/76467e8a3c46aca3687ffcfe964a2889.svg
Fanchao Qi
user
fanchao-qi
671742ab0c095b8e5d099559
Baobao Chang
671742ab0c095b8e5d09955a
Maosong Sun
Selecting Influential Samples for Long Context Alignment via Homologous
  Models' Guidance and Contextual Awareness Measurement
The expansion of large language models to effectively handle instructions
with extremely long contexts has yet to be fully investigated. The primary
obstacle lies in constructing a high-quality long instruction-following dataset
devised for long context alignment. Existing studies have attempted to scale up
the available data volume by synthesizing long instruction-following samples.
However, indiscriminately increasing the quantity of data without a
well-defined strategy for ensuring data quality may introduce low-quality
samples and restrict the final performance. To bridge this gap, we aim to
address the unique challenge of long-context alignment, i.e., modeling the
long-range dependencies for handling instructions and lengthy input contexts.
We propose GATEAU, a novel framework designed to identify the influential and
high-quality samples enriched with long-range dependency relations by utilizing
crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
(CAM). Specifically, HMG attempts to measure the difficulty of generating
corresponding responses due to the long-range dependencies, using the
perplexity scores of the response from two homologous models with different
context windows. Also, the role of CAM is to measure the difficulty of
understanding the long input contexts due to long-range dependencies by
evaluating whether the model's attention is focused on important segments.
Built upon both proposed methods, we select the most challenging samples as the
influential data to effectively frame the long-range dependencies, thereby
achieving better performance of LLMs. Comprehensive experiments indicate that
GATEAU effectively identifies samples enriched with long-range dependency
relations and the model trained on these selected samples exhibits better
instruction-following and long-context understanding capabilities.
GATEAU framework improves large language models' performance with long contexts by identifying high-quality samples with long-range dependencies using Homologous Models' Guidance and Contextual Awareness Measurement.
large language models long instruction-following dataset long-range dependencies GATEAU Homologous Models' Guidance Contextual Awareness Measurement perplexity scores attention focus