Revisiting Long-context Modeling from Context Denoising Perspective
Long-context models (LCMs) have demonstrated great potential in processing
long sequences, facilitating many real-world applications. The success of LCMs
can be attributed to their ability to locate implicit critical information
within the context for further prediction. However, recent research reveals
that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens,
that can mislead model attention. In this paper, we conduct a fine-grained
analysis of the context noise and propose an effective metric, the Integrated
Gradient (IG) score, to detect and quantify the noise information within the
context. Our findings reveal that even simple mitigation of detected context
noise can substantially boost the model's attention on critical tokens and
benefit subsequent predictions. Building on this insight, we propose Context
Denoising Training (CDT), a straightforward yet effective training strategy
that improves attention on critical tokens while reinforcing their influence on
model predictions. Extensive experiments across four tasks, under both context
window scaling and long-context alignment settings, demonstrate the superiority
of CDT. Notably, when trained with CDT, an open-source 8B model can achieve
performance (50.92) comparable to GPT-4o (51.00).
68e64fcf975ac4c405ef2263
Zecheng Tang
claimed_verified
2025-10-10T09:38:02.076Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
user
ZetangForward
68e64fcf975ac4c405ef2264
Baibei Ji
admin_assigned
2025-10-23T23:10:45.873Z
65731fc31345577b7071d7df
https://cdn-avatars.huggingface.co/v1/production/uploads/65731fc31345577b7071d7df/T8qnqIxnycvy1AP8CKAPb.png
Baibei Ji
user
iiiiGray
68e64fcf975ac4c405ef2265
Juntao Li
68e64fcf975ac4c405ef2266
Lijun Wu
68e64fcf975ac4c405ef2267
Haijia Gui
68e64fcf975ac4c405ef2268
Min Zhang
Revisiting Long-context Modeling from Context Denoising Perspective
Long-context models (LCMs) have demonstrated great potential in processing
long sequences, facilitating many real-world applications. The success of LCMs
can be attributed to their ability to locate implicit critical information
within the context for further prediction. However, recent research reveals
that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens,
that can mislead model attention. In this paper, we conduct a fine-grained
analysis of the context noise and propose an effective metric, the Integrated
Gradient (IG) score, to detect and quantify the noise information within the
context. Our findings reveal that even simple mitigation of detected context
noise can substantially boost the model's attention on critical tokens and
benefit subsequent predictions. Building on this insight, we propose Context
Denoising Training (CDT), a straightforward yet effective training strategy
that improves attention on critical tokens while reinforcing their influence on
model predictions. Extensive experiments across four tasks, under both context
window scaling and long-context alignment settings, demonstrate the superiority
of CDT. Notably, when trained with CDT, an open-source 8B model can achieve
performance (50.92) comparable to GPT-4o (51.00).
Context Denoising Training (CDT) improves long-context models' performance by mitigating contextual noise and enhancing attention on critical tokens.
long-context models contextual noise Integrated Gradient (IG) score Context Denoising Training (CDT) context window scaling long-context alignment