HoPE: Hybrid of Position Embedding for Length Generalization in
  Vision-Language Models
Vision-Language Models (VLMs) have made significant progress in multimodal
tasks. However, their performance often deteriorates in long-context scenarios,
particularly long videos. While Rotary Position Embedding (RoPE) has been
widely adopted for length generalization in Large Language Models (LLMs),
extending vanilla RoPE to capture the intricate spatial-temporal dependencies
in videos remains an unsolved challenge. Existing methods typically allocate
different frequencies within RoPE to encode 3D positional information. However,
these allocation strategies mainly rely on heuristics, lacking in-depth
theoretical analysis. In this paper, we first study how different allocation
strategies impact the long-context capabilities of VLMs. Our analysis reveals
that current multimodal RoPEs fail to reliably capture semantic similarities
over extended contexts. To address this issue, we propose HoPE, a Hybrid of
Position Embedding designed to improve the long-context capabilities of VLMs.
HoPE introduces a hybrid frequency allocation strategy for reliable semantic
modeling over arbitrarily long context, and a dynamic temporal scaling
mechanism to facilitate robust learning and flexible inference across diverse
context lengths. Extensive experiments across four video benchmarks on long
video understanding and retrieval tasks demonstrate that HoPE consistently
outperforms existing methods, confirming its effectiveness. Code is available
at https://github.com/hrlics/HoPE.
68385abc8c8e3b721702d006
Haoran Li
claimed_verified
2025-05-29T20:19:23.117Z
64e86d6e7ef2c72cbcdb27b7
https://cdn-avatars.huggingface.co/v1/production/uploads/64e86d6e7ef2c72cbcdb27b7/RLuJJ1xuO665LMWneVDZz.jpeg
Haoran Li
user
brian13
68385abc8c8e3b721702d007
Yingjie Qin
68385abc8c8e3b721702d008
Baoyuan Ou
68385abc8c8e3b721702d009
Lai Xu
68385abc8c8e3b721702d00a
Ruiwen Xu
HoPE: Hybrid of Position Embedding for Length Generalization in
  Vision-Language Models
Vision-Language Models (VLMs) have made significant progress in multimodal
tasks. However, their performance often deteriorates in long-context scenarios,
particularly long videos. While Rotary Position Embedding (RoPE) has been
widely adopted for length generalization in Large Language Models (LLMs),
extending vanilla RoPE to capture the intricate spatial-temporal dependencies
in videos remains an unsolved challenge. Existing methods typically allocate
different frequencies within RoPE to encode 3D positional information. However,
these allocation strategies mainly rely on heuristics, lacking in-depth
theoretical analysis. In this paper, we first study how different allocation
strategies impact the long-context capabilities of VLMs. Our analysis reveals
that current multimodal RoPEs fail to reliably capture semantic similarities
over extended contexts. To address this issue, we propose HoPE, a Hybrid of
Position Embedding designed to improve the long-context capabilities of VLMs.
HoPE introduces a hybrid frequency allocation strategy for reliable semantic
modeling over arbitrarily long context, and a dynamic temporal scaling
mechanism to facilitate robust learning and flexible inference across diverse
context lengths. Extensive experiments across four video benchmarks on long
video understanding and retrieval tasks demonstrate that HoPE consistently
outperforms existing methods, confirming its effectiveness. Code is available
at https://github.com/hrlics/HoPE.
HoPE, a Hybrid of Position Embedding, enhances VLMs' long-context performance in videos through improved frequency allocation and dynamic temporal scaling.
Rotary Position Embedding RoPE Large Language Models LLMs multimodal RoPEs HoPE Hybrid of Position Embedding frequency allocation dynamic temporal scaling video benchmarks long video understanding retrieval tasks