AerialMegaDepth: Learning Aerial-Ground Reconstruction and View
  Synthesis
We explore the task of geometric reconstruction of images captured from a
mixture of ground and aerial views. Current state-of-the-art learning-based
approaches fail to handle the extreme viewpoint variation between aerial-ground
image pairs. Our hypothesis is that the lack of high-quality, co-registered
aerial-ground datasets for training is a key reason for this failure. Such data
is difficult to assemble precisely because it is difficult to reconstruct in a
scalable way. To overcome this challenge, we propose a scalable framework
combining pseudo-synthetic renderings from 3D city-wide meshes (e.g., Google
Earth) with real, ground-level crowd-sourced images (e.g., MegaDepth). The
pseudo-synthetic data simulates a wide range of aerial viewpoints, while the
real, crowd-sourced images help improve visual fidelity for ground-level images
where mesh-based renderings lack sufficient detail, effectively bridging the
domain gap between real images and pseudo-synthetic renderings. Using this
hybrid dataset, we fine-tune several state-of-the-art algorithms and achieve
significant improvements on real-world, zero-shot aerial-ground tasks. For
example, we observe that baseline DUSt3R localizes fewer than 5% of
aerial-ground pairs within 5 degrees of camera rotation error, while
fine-tuning with our data raises accuracy to nearly 56%, addressing a major
failure point in handling large viewpoint changes. Beyond camera estimation and
scene reconstruction, our dataset also improves performance on downstream tasks
like novel-view synthesis in challenging aerial-ground scenarios, demonstrating
the practical value of our approach in real-world applications.
6804392129303a3402c4f38e
Khiem Vuong
claimed_verified
2025-04-20T15:01:35.224Z
631bfb21f6bc4be4a6592afc
https://cdn-avatars.huggingface.co/v1/production/uploads/631bfb21f6bc4be4a6592afc/FRgc7nwHylQZ9QURrr88y.jpeg
Khiem Vuong
user
kvuong2711
6804392129303a3402c4f38f
Anurag Ghosh
6804392129303a3402c4f390
Deva Ramanan
6804392129303a3402c4f391
Srinivasa Narasimhan
6804392129303a3402c4f392
Shubham Tulsiani
AerialMegaDepth: Learning Aerial-Ground Reconstruction and View
  Synthesis
We explore the task of geometric reconstruction of images captured from a
mixture of ground and aerial views. Current state-of-the-art learning-based
approaches fail to handle the extreme viewpoint variation between aerial-ground
image pairs. Our hypothesis is that the lack of high-quality, co-registered
aerial-ground datasets for training is a key reason for this failure. Such data
is difficult to assemble precisely because it is difficult to reconstruct in a
scalable way. To overcome this challenge, we propose a scalable framework
combining pseudo-synthetic renderings from 3D city-wide meshes (e.g., Google
Earth) with real, ground-level crowd-sourced images (e.g., MegaDepth). The
pseudo-synthetic data simulates a wide range of aerial viewpoints, while the
real, crowd-sourced images help improve visual fidelity for ground-level images
where mesh-based renderings lack sufficient detail, effectively bridging the
domain gap between real images and pseudo-synthetic renderings. Using this
hybrid dataset, we fine-tune several state-of-the-art algorithms and achieve
significant improvements on real-world, zero-shot aerial-ground tasks. For
example, we observe that baseline DUSt3R localizes fewer than 5% of
aerial-ground pairs within 5 degrees of camera rotation error, while
fine-tuning with our data raises accuracy to nearly 56%, addressing a major
failure point in handling large viewpoint changes. Beyond camera estimation and
scene reconstruction, our dataset also improves performance on downstream tasks
like novel-view synthesis in challenging aerial-ground scenarios, demonstrating
the practical value of our approach in real-world applications.
A scalable hybrid dataset combining pseudo-synthetic renderings and real ground-level images improves performance in geometric reconstruction and novel-view synthesis across aerial-ground viewpoints.
geometric reconstruction aerial-ground views viewpoint variation co-registered datasets pseudo-synthetic renderings 3D city-wide meshes crowd-sourced images visual fidelity domain gap baseline DUSt3R camera rotation error novel-view synthesis