UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence
  with Spatial Reasoning and Understanding
Urban research involves a wide range of scenarios and tasks that require the
understanding of multi-modal data. Current methods often focus on specific data
types and lack a unified framework in urban field for processing them
comprehensively. The recent success of multi-modal large language models
(MLLMs) presents a promising opportunity to overcome this limitation. In this
paper, we introduce UrbanLLaVA, a multi-modal large language model
designed to process these four types of data simultaneously and achieve strong
performance across diverse urban tasks compared with general MLLMs. In
UrbanLLaVA, we first curate a diverse urban instruction dataset
encompassing both single-modal and cross-modal urban data, spanning from
location view to global view of urban environment. Additionally, we propose a
multi-stage training framework that decouples spatial reasoning enhancement
from domain knowledge learning, thereby improving the compatibility and
downstream performance of UrbanLLaVA across diverse urban tasks.
Finally, we also extend existing benchmark for urban research to assess the
performance of MLLMs across a wide range of urban tasks. Experimental results
from three cities demonstrate that UrbanLLaVA outperforms
open-source and proprietary MLLMs in both single-modal tasks and complex
cross-modal tasks and shows robust generalization abilities across cities.
Source codes and data are openly accessible to the research community via
https://github.com/tsinghua-fib-lab/UrbanLLaVA.
686376f6588cea0da970c92b
Jie Feng
claimed_verified
2025-07-01T10:13:14.367Z
6465d3bd63e7e09dd02e95c3
/avatars/b2798bd5f8368f956bf7fab79d9432f0.svg
Jie Feng
user
JJ-TMT
686376f6588cea0da970c92c
Shengyuan Wang
686376f6588cea0da970c92d
Tianhui Liu
686376f6588cea0da970c92e
Yanxin Xi
686376f6588cea0da970c92f
Yong Li
UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence
  with Spatial Reasoning and Understanding
Urban research involves a wide range of scenarios and tasks that require the
understanding of multi-modal data. Current methods often focus on specific data
types and lack a unified framework in urban field for processing them
comprehensively. The recent success of multi-modal large language models
(MLLMs) presents a promising opportunity to overcome this limitation. In this
paper, we introduce UrbanLLaVA, a multi-modal large language model
designed to process these four types of data simultaneously and achieve strong
performance across diverse urban tasks compared with general MLLMs. In
UrbanLLaVA, we first curate a diverse urban instruction dataset
encompassing both single-modal and cross-modal urban data, spanning from
location view to global view of urban environment. Additionally, we propose a
multi-stage training framework that decouples spatial reasoning enhancement
from domain knowledge learning, thereby improving the compatibility and
downstream performance of UrbanLLaVA across diverse urban tasks.
Finally, we also extend existing benchmark for urban research to assess the
performance of MLLMs across a wide range of urban tasks. Experimental results
from three cities demonstrate that UrbanLLaVA outperforms
open-source and proprietary MLLMs in both single-modal tasks and complex
cross-modal tasks and shows robust generalization abilities across cities.
Source codes and data are openly accessible to the research community via
https://github.com/tsinghua-fib-lab/UrbanLLaVA.
UrbanLLaVA, a multi-modal large language model, processes diverse urban data types and outperforms existing models in various urban tasks.
multi-modal large language models MLLMs UrbanLLaVA urban instruction dataset multi-stage training framework spatial reasoning domain knowledge learning benchmark urban tasks single-modal tasks cross-modal tasks robust generalization