NeRF for unconstrained satellite imagery reconstruction

ABSTRACT The neural radiance fields (NeRF) has achieved impressive performance in three-dimensional (3D) scene reconstruction. However, due to the varying illumination and transient occluders, the synthesized images from unseen solar directions are unrealistic and the digital surface models (DSMs) have blurred edges and irregular surfaces. To address these problems, this paper presents a NeRF for unconstrained satellite imagery reconstruction, dubbed as NeRF-U. NeRF-U jointly estimates illumination, appearance, and geometry using multi-date and multi-view image collections for a given scene, leveraging a physically based rendering technique to produce rendered images and DSMs. The spatially varying illumination is modelled using an incoming light model that consists of geometry-related solar visibility, learnable skylight, and normalized sunlight. In addition, an image-dependent, image-coordinate-based multi-layer perceptron (MLP) accurately decomposes the transient and static components of the scene, which can be removed from the network architecture during testing. Our NeRF-U outperforms previous methods on novel view synthesis and digital surface modelling, demonstrating the significant potential in unconstrained satellite imagery reconstruction.