Unleashing the Potential of Multimodal LLMs for Zero-Shot
  Spatio-Temporal Video Grounding
Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal
tube of a video, as specified by the input text query. In this paper, we
utilize multimodal large language models (MLLMs) to explore a zero-shot
solution in STVG. We reveal two key insights about MLLMs: (1) MLLMs tend to
dynamically assign special tokens, referred to as grounding tokens,
for grounding the text query; and (2) MLLMs often suffer from suboptimal
grounding due to the inability to fully integrate the cues in the text query
(e.g., attributes, actions) for inference. Based on these insights, we
propose a MLLM-based zero-shot framework for STVG, which includes novel
decomposed spatio-temporal highlighting (DSTH) and temporal-augmented
assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH
strategy first decouples the original query into attribute and action
sub-queries for inquiring the existence of the target both spatially and
temporally. It then uses a novel logit-guided re-attention (LRA) module to
learn latent variables as spatial and temporal prompts, by regularizing token
predictions for each sub-query. These prompts highlight attribute and action
cues, respectively, directing the model's attention to reliable spatial and
temporal related visual regions. In addition, as the spatial grounding by the
attribute sub-query should be temporally consistent, we introduce the TAS
strategy to assemble the predictions using the original video frames and the
temporal-augmented frames as inputs to help improve temporal consistency. We
evaluate our method on various MLLMs, and show that it outperforms SOTA methods
on three common STVG benchmarks.
  The code will be available at https://github.com/zaiquanyang/LLaVA_Next_STVG.
68ccc1663df9ac65e93dc664
Zaiquan Yang
admin_assigned
2025-09-19T10:37:06.882Z
629c73c588987318a42580a1
/avatars/05ac913fcf7d94d51b75a3fea87d6de2.svg
Zaiquan yang
user
zaiquan
68ccc1663df9ac65e93dc665
Yuhao Liu
admin_assigned
2025-09-19T10:36:54.136Z
66725eb419bebc69b5d7ad05
/avatars/36218f1f19b84c7dfbe8912a620a9dd6.svg
yuhaoliu
user
yuhaoliu
68ccc1663df9ac65e93dc666
Gerhard Hancke
68ccc1663df9ac65e93dc667
Rynson W. H. Lau
Unleashing the Potential of Multimodal LLMs for Zero-Shot
  Spatio-Temporal Video Grounding
Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal
tube of a video, as specified by the input text query. In this paper, we
utilize multimodal large language models (MLLMs) to explore a zero-shot
solution in STVG. We reveal two key insights about MLLMs: (1) MLLMs tend to
dynamically assign special tokens, referred to as grounding tokens,
for grounding the text query; and (2) MLLMs often suffer from suboptimal
grounding due to the inability to fully integrate the cues in the text query
(e.g., attributes, actions) for inference. Based on these insights, we
propose a MLLM-based zero-shot framework for STVG, which includes novel
decomposed spatio-temporal highlighting (DSTH) and temporal-augmented
assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH
strategy first decouples the original query into attribute and action
sub-queries for inquiring the existence of the target both spatially and
temporally. It then uses a novel logit-guided re-attention (LRA) module to
learn latent variables as spatial and temporal prompts, by regularizing token
predictions for each sub-query. These prompts highlight attribute and action
cues, respectively, directing the model's attention to reliable spatial and
temporal related visual regions. In addition, as the spatial grounding by the
attribute sub-query should be temporally consistent, we introduce the TAS
strategy to assemble the predictions using the original video frames and the
temporal-augmented frames as inputs to help improve temporal consistency. We
evaluate our method on various MLLMs, and show that it outperforms SOTA methods
on three common STVG benchmarks.
  The code will be available at https://github.com/zaiquanyang/LLaVA_Next_STVG.
A zero-shot framework using multimodal large language models for spatio-temporal video grounding employs decomposed spatio-temporal highlighting and temporal-augmented assembling strategies to improve grounding accuracy.
spatio-temporal video grounding multimodal large language models zero-shot solution grounding tokens decomposed spatio-temporal highlighting temporal-augmented assembling logit-guided re-attention latent variables spatial prompts temporal prompts temporal consistency