LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
Reasoning is a fundamental capability for solving complex multi-step
problems, particularly in visual contexts where sequential step-wise
understanding is essential. Existing approaches lack a comprehensive framework
for evaluating visual reasoning and do not emphasize step-wise problem-solving.
To this end, we propose a comprehensive framework for advancing step-by-step
visual reasoning in large language models (LMMs) through three key
contributions. First, we introduce a visual reasoning benchmark specifically
designed to evaluate multi-step reasoning tasks. The benchmark presents a
diverse set of challenges with eight different categories ranging from complex
visual perception to scientific reasoning with over 4k reasoning steps in
total, enabling robust evaluation of LLMs' abilities to perform accurate and
interpretable visual reasoning across multiple steps. Second, we propose a
novel metric that assesses visual reasoning quality at the granularity of
individual steps, emphasizing both correctness and logical coherence. The
proposed metric offers deeper insights into reasoning performance compared to
traditional end-task accuracy metrics. Third, we present a new multimodal
visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum
learning approach, where tasks are progressively organized to facilitate
incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is
designed for multi-step reasoning and learns step-by-step through a structured
training paradigm. Extensive experiments show that our LlamaV-o1 outperforms
existing open-source models and performs favorably against close-source
proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an
average score of 67.3 with an absolute gain of 3.8\% across six benchmarks
while being 5 times faster during inference scaling. Our benchmark, model, and
code are publicly available.
6784817c956a6e09449547bd
Omkar Thawakar
admin_assigned
2025-01-13T09:25:42.768Z
64b7d2ad8c632fbca9507431
/avatars/76c31ea218108cf6c3715269f7605404.svg
Omkar Thawakar
user
omkarthawakar
6784817c956a6e09449547be
Dinura Dissanayake
claimed_verified
2025-02-04T09:39:49.771Z
6639cd849662bb58d5fe5793
https://cdn-avatars.huggingface.co/v1/production/uploads/6639cd849662bb58d5fe5793/PHUa38L5_llipN52ngajY.png
Dissanayake
user
Dinura
6784817c956a6e09449547bf
Ketan More
6784817c956a6e09449547c0
Ritesh Thawkar
admin_assigned
2025-01-13T09:26:18.535Z
664cec3ba016876f526afe7a
/avatars/69007f7e7f7ff78069101deb4b354e37.svg
Ritesh Thawkar
user
Ritesh-hf
6784817c956a6e09449547c1
Ahmed Heakl
claimed_verified
2025-01-13T08:54:30.443Z
656864e12d73834278a8dea7
https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
Ahmed Heakl
user
ahmedheakl
6784817c956a6e09449547c2
Noor Ahsan
6784817c956a6e09449547c3
Yuhao Li
claimed_verified
2025-03-10T08:04:27.329Z
667840df246665be1af09cb5
/avatars/c714d3fbb32611704effc7964a248edc.svg
Yuhao Li
user
UHow
6784817c956a6e09449547c4
Mohammed Zumri
6784817c956a6e09449547c5
Jean Lahoud
6784817c956a6e09449547c6
Rao Muhammad Anwer
6784817c956a6e09449547c7
Hisham Cholakkal
admin_assigned
2025-01-13T09:27:47.044Z
654a5f4f9b8bd6406d45bb46
/avatars/ac0d7eef62cd98a280b162cf7896b1a2.svg
Hisham Cholakkal
user
hishamcholakkal
6784817c956a6e09449547c8
Ivan Laptev
6784817c956a6e09449547c9
Mubarak Shah
6784817c956a6e09449547ca
Fahad Shahbaz Khan
6784817c956a6e09449547cb
Salman Khan
LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
Reasoning is a fundamental capability for solving complex multi-step
problems, particularly in visual contexts where sequential step-wise
understanding is essential. Existing approaches lack a comprehensive framework
for evaluating visual reasoning and do not emphasize step-wise problem-solving.
To this end, we propose a comprehensive framework for advancing step-by-step
visual reasoning in large language models (LMMs) through three key
contributions. First, we introduce a visual reasoning benchmark specifically
designed to evaluate multi-step reasoning tasks. The benchmark presents a
diverse set of challenges with eight different categories ranging from complex
visual perception to scientific reasoning with over 4k reasoning steps in
total, enabling robust evaluation of LLMs' abilities to perform accurate and
interpretable visual reasoning across multiple steps. Second, we propose a
novel metric that assesses visual reasoning quality at the granularity of
individual steps, emphasizing both correctness and logical coherence. The
proposed metric offers deeper insights into reasoning performance compared to
traditional end-task accuracy metrics. Third, we present a new multimodal
visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum
learning approach, where tasks are progressively organized to facilitate
incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is
designed for multi-step reasoning and learns step-by-step through a structured
training paradigm. Extensive experiments show that our LlamaV-o1 outperforms
existing open-source models and performs favorably against close-source
proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an
average score of 67.3 with an absolute gain of 3.8\% across six benchmarks
while being 5 times faster during inference scaling. Our benchmark, model, and
code are publicly available.
A framework for evaluating and improving step-by-step visual reasoning in large language models using a specialized benchmark and a novel multimodal model trained with curriculum learning.
visual reasoning large language models visual reasoning benchmark multi-step reasoning tasks reasoning steps visual perception scientific reasoning visual reasoning quality multimodal visual reasoning model LlamaV-o1 curriculum learning incremental skill acquisition problem-solving Llava-CoT end-task accuracy metrics