LLaVAction: evaluating and training multi-modal large language models
  for action recognition
Understanding human behavior requires measuring behavioral actions. Due to
its complexity, behavior is best mapped onto a rich, semantic structure such as
language. The recent development of multi-modal large language models (MLLMs)
is a promising candidate for a wide range of action understanding tasks. In
this work, we focus on evaluating and then improving MLLMs to perform action
recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most
challenging egocentric action datasets, to the form of video multiple question
answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult
incorrect answers as distractors, leading MLLMs struggle to recognize the
correct actions. We propose a series of methods that greatly improve the MLLMs'
ability to perform action recognition, achieving state-of-the-art on both the
EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points
in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other
action-related video benchmarks such as EgoSchema, PerceptionTest,
LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising
path forward for complex action tasks. Code and models are available at:
https://github.com/AdaptiveMotorControlLab/LLaVAction.
67e47a92d762ffface602862
Shaokai Ye
67e47a92d762ffface602863
Haozhe Qi
67e47a92d762ffface602864
Alexander Mathis
67e47a92d762ffface602865
Mackenzie W. Mathis
extracted_confirmed
2025-03-26T22:13:22.189Z
62c2e4d672ff397f721ac6f7
https://cdn-avatars.huggingface.co/v1/production/uploads/1656944910507-62c2e4d672ff397f721ac6f7.jpeg
Mackenzie Mathis
user
mwmathis
LLaVAction: evaluating and training multi-modal large language models
  for action recognition
Understanding human behavior requires measuring behavioral actions. Due to
its complexity, behavior is best mapped onto a rich, semantic structure such as
language. The recent development of multi-modal large language models (MLLMs)
is a promising candidate for a wide range of action understanding tasks. In
this work, we focus on evaluating and then improving MLLMs to perform action
recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most
challenging egocentric action datasets, to the form of video multiple question
answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult
incorrect answers as distractors, leading MLLMs struggle to recognize the
correct actions. We propose a series of methods that greatly improve the MLLMs'
ability to perform action recognition, achieving state-of-the-art on both the
EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points
in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other
action-related video benchmarks such as EgoSchema, PerceptionTest,
LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising
path forward for complex action tasks. Code and models are available at:
https://github.com/AdaptiveMotorControlLab/LLaVAction.
Multi-modal large language models enhanced with question-answering methods improve action recognition accuracy on several benchmarks, including EPIC-KITCHENS-100 and others.
multi-modal large language models MLLMs EPIC-KITCHENS-100 video multiple question answering EPIC-KITCHENS-100-MQA GPT-4o EgoSchema PerceptionTest LongVideoBench VideoMME MVBench