Towards foundational LiDAR world models with efficient latent flow matching
Tianran Liu, Shengwen Zhao, Nicholas Rhinehart
LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain it was built. We conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse-beam \& dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of finetuning data, our experiments show that a single pre-trained model can bring up to 11\% absolute (83\% rel.) improvement over from-scratch training. This transferability of dynamic learning significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method achieves state-of-the-art forecasting performance using only 5\% of the labeled training data required by prior models. We also observed inefficiencies of current LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objective. To address this, we propose a latent flow matching (CFM)-based approach that achieves state-of-the-art reconstruction accuracy using only half the training data and a 6x compression ratio compared to prior methods. Based this compact latent, our model achieves SOTA performance on future-trajectory-conditioned semantic occupancy forecasting while being 23x more computationally efficient (a 28x FPS speedup); and achieves SOTA performance on semantic occupancy forecasting while being 2x more computationally efficient (a 1.1x FPS speedup).