Enhanced Stable View Synthesis

Nishant Jain1*
Suryansh Kumar2*†
Luc Van Gool2, 3

IIT Roorkee1, ETH Zürich2, KU Lueven3
*Equal contribution, †Corresponding Author

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.




Abstract
We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging, leading to inferior results using the state-of-the-art stable view synthesis (SVS) method. SVS and related methods fail for outdoor scenes primarily due to (i) over-relying on the multiview stereo (MVS) for geometric scaffold recovery and (ii) assuming COLMAP computed camera poses as the best possible estimates, despite it being well-studied that MVS 3D reconstruction accuracy is limited to scene disparity and camera-pose accuracy is sensitive to key-point correspondence selection. This work proposes a principled way to enhance novel view synthesis solutions drawing inspiration from the basics of multiple view geometry. By leveraging the complementary behavior of MVS and monocular depth, we arrive at a better scene depth per view for nearby and far points, respectively. Moreover, our approach jointly refines camera poses with image-based rendering via multiple rotation averaging graph optimization. The recovered scene depth and the camera-pose help better view-dependent on-surface feature aggregation of the entire scene. Extensive evaluation of our approach on the popular benchmark dataset, such as Tanks and Temples, shows substantial improvement in view synthesis results compared to the prior art. For instance, our method shows 1.5 dB of PSNR improvement on the Tank and Temples. Similar statistics are observed when tested on other benchmark datasets such as FVS, Mip-NeRF 360, and DTU.




Paper

Enhanced Stable View Synthesis

Nishant Jain*, Suryansh Kumar*†, Luc Van Gool.

IEEE/CVF CVPR 2023, Vancouver, Canada.

[Paper]
[Supp]
[Bibtex]


Results

3D Reconstruction

Rendered Novel View



Qualitative Comparison Video





Authors

Nishant Jain

Suryansh Kumar

Luc Van Gool




Copyright © 2023 Suryansh Kumar