GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Recent advances in large multimodal models (LMMs) have recognized
fine-grained grounding as an imperative factor of visual understanding and
dialogue. However, the benefits of such representation in LMMs are limited to
the natural image domain, and these models perform poorly for remote sensing
(RS). The distinct overhead viewpoint, scale variation, and presence of small
objects in high-resolution RS imagery present a unique challenge in
region-level comprehension. Moreover, the development of the grounding
conversation capability of LMMs within RS is hindered by the lack of granular,
RS domain-specific grounded data. Addressing these limitations, we propose
GeoPixel - the first end-to-end high resolution RS-LMM that supports
pixel-level grounding. This capability allows fine-grained visual perception by
generating interleaved masks in conversation. GeoPixel supports up to 4K HD
resolution in any aspect ratio, ideal for high-precision RS image analysis. To
support the grounded conversation generation (GCG) in RS imagery, we curate a
visually grounded dataset GeoPixelD through a semi-automated pipeline that
utilizes set-of-marks prompting and spatial priors tailored for RS data to
methodically control the data generation process. GeoPixel demonstrates
superior performance in pixel-level comprehension, surpassing existing LMMs in
both single-target and multi-target segmentation tasks. Our methodological
ablation studies validate the effectiveness of each component in the overall
architecture. Our code and data will be publicly released.
679789af4a10be7109a28675
Akashah Shabbir
679789af4a10be7109a28676
Mohammed Zumri
679789af4a10be7109a28677
Mohammed Bennamoun
679789af4a10be7109a28678
Fahad S. Khan
679789af4a10be7109a28679
Salman Khan
GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Recent advances in large multimodal models (LMMs) have recognized
fine-grained grounding as an imperative factor of visual understanding and
dialogue. However, the benefits of such representation in LMMs are limited to
the natural image domain, and these models perform poorly for remote sensing
(RS). The distinct overhead viewpoint, scale variation, and presence of small
objects in high-resolution RS imagery present a unique challenge in
region-level comprehension. Moreover, the development of the grounding
conversation capability of LMMs within RS is hindered by the lack of granular,
RS domain-specific grounded data. Addressing these limitations, we propose
GeoPixel - the first end-to-end high resolution RS-LMM that supports
pixel-level grounding. This capability allows fine-grained visual perception by
generating interleaved masks in conversation. GeoPixel supports up to 4K HD
resolution in any aspect ratio, ideal for high-precision RS image analysis. To
support the grounded conversation generation (GCG) in RS imagery, we curate a
visually grounded dataset GeoPixelD through a semi-automated pipeline that
utilizes set-of-marks prompting and spatial priors tailored for RS data to
methodically control the data generation process. GeoPixel demonstrates
superior performance in pixel-level comprehension, surpassing existing LMMs in
both single-target and multi-target segmentation tasks. Our methodological
ablation studies validate the effectiveness of each component in the overall
architecture. Our code and data will be publicly released.
GeoPixel, an end-to-end RS-LMM, achieves superior pixel-level comprehension in remote sensing images through fine-grained grounding and interleaved mask generation in conversations.
large multimodal models fine-grained grounding visual understanding dialogue remote sensing region-level comprehension grounded conversation capability high-resolution RS imagery pixel-level grounding interleaved masks high-precision RS image analysis GeoPixel GeoPixelD set-of-marks prompting spatial priors superior performance single-target segmentation multi-target segmentation