AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via
  GRPO
Large Language Models (LLMs) have demonstrated impressive capabilities in
language processing, yet they often struggle with tasks requiring genuine
visual spatial reasoning. In this paper, we introduce a novel two-stage
training framework designed to equip standard LLMs with visual reasoning
abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT)
on a curated dataset of tokenized maze representations to teach the model to
predict step-by-step movement commands. Next, we apply Group Relative Policy
Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted
reward function to refine the model's sequential decision-making and encourage
emergent chain-of-thought behaviors. Experimental results on synthetically
generated mazes show that while a baseline model fails to navigate the maze,
the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning
boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more
robust and self-corrective reasoning, highlighting the potential of our
approach to bridge the gap between language models and visual spatial tasks.
These findings offer promising implications for applications in robotics,
autonomous navigation, and other domains that require integrated visual and
sequential reasoning.
67b7eeddaf9f1b1bd95b878b
Alan Dao
admin_assigned
2025-02-21T15:03:59.165Z
62d7b2339b629105a5d6888a
https://cdn-avatars.huggingface.co/v1/production/uploads/62d7b2339b629105a5d6888a/pYi994fqzXctgoed79oUo.png
Alan Dao
user
alandao
67b7eeddaf9f1b1bd95b878c
Dinh Bach Vu
AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via
  GRPO
Large Language Models (LLMs) have demonstrated impressive capabilities in
language processing, yet they often struggle with tasks requiring genuine
visual spatial reasoning. In this paper, we introduce a novel two-stage
training framework designed to equip standard LLMs with visual reasoning
abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT)
on a curated dataset of tokenized maze representations to teach the model to
predict step-by-step movement commands. Next, we apply Group Relative Policy
Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted
reward function to refine the model's sequential decision-making and encourage
emergent chain-of-thought behaviors. Experimental results on synthetically
generated mazes show that while a baseline model fails to navigate the maze,
the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning
boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more
robust and self-corrective reasoning, highlighting the potential of our
approach to bridge the gap between language models and visual spatial tasks.
These findings offer promising implications for applications in robotics,
autonomous navigation, and other domains that require integrated visual and
sequential reasoning.
A novel two-stage training framework equips LLMs with visual reasoning by first using Supervised Fine Tuning on maze representations and then Group Relative Policy Optimization for refined decision-making, significantly improving maze navigation accuracy.
Supervised Fine Tuning (SFT) Group Relative Policy Optimization (GRPO) tokenized maze representations sequential decision-making chain-of-thought behaviors maze navigation accuracy robust reasoning self-corrective reasoning