Project Image Semantically-Aware Aerial Reconstruction from Multi-Modal Data
We consider a methodology for integrating multiple sensors along with semantic information to enhance scene representations. We propose a probabilistic generative model for inferring semantically-informed aerial reconstructions from multi-modal data within a consistent mathematical framework. The approach, called Semantically Aware Aerial Reconstruction (SAAR), not only exploits inferred scene geometry, appearance, and semantic observations to obtain a meaningful categorization of the data, but also extends previously proposed methods by imposing structure on the prior over geometry, appearance, and semantic labels. This leads to more accurate reconstructions and the ability to fill in missing contextual labels via joint sensor and semantic information. We introduce a new multi-modal synthetic dataset in order to provide quantitative performance analysis. Additionally, we apply the model to real-world data and exploit OpenStreetMap as a source of semantic observations. We show quantitative improvements in reconstruction accuracy of large-scale urban scenes from the combination of LiDAR, aerial photography, and semantic data. Furthermore, we demonstrate the model's ability to fill in for missing sensed data, leading to more interpretable reconstructions.

People Involved: Randi Cabezas, Julian Straub, John W. Fisher III

Data now available! (Download from here )

In the following video, we briefly describe the method and highlight the main advantages of the proposed approach. We also showcase typical results.

Paper, supplemental materials and code can be found here.


JW Player not found
If you are having trouble playing the embedded video, it can be downloaded from here (right click and save-as).