Non-Rigid Structure from Motion

Abstract

The problem of recovering the 3D shape of a non-rigidly deforming object from its image feature correspondences across multiple frames is widely known as Non-Rigid Structure from Motion (NRSfM). It is a well-defined classical problem whose solution can assist several industrial applications in virtual reality, medical surgery, movies, etc. However, the task is challenging due to the inherent unconstrained nature of the problem itself, as many 3D varying configurations can have similar image projections. To date, no algorithm can solve NRSfM for all kinds of conceivable motion. Consequently, additional constraints, priors, and assumptions are often employed to solve NRSfM. Our work takes on this challenging problem and proposes algorithms that have set a new performance benchmark to solve NRSfM. Our solutions discuss the classical work in NRSfM and suggest necessary elementary modifications to it. The foundation of our contributions surpasses the traditional single object NRSFM and, for the first time, provides an effective formulation to realize multi-body NRSfM. Further, most NRSfM factorization methods can effectively handle only sparse feature correspondences. Later, sparse 3d points are used to approximate the object's dense shape via the organization of 3d points, planes, or other elementary geometric primitive. Nevertheless, sparse representation provides incomplete information about the object's global shape. On the contrary, our proposed methods can directly solve dense NRSfM showing state-of-the-art accuracy.

For dense NRSfM, we show that we can recover deforming shape 3d with favorable accuracy using just matrix factorization approach. However, a carefully modeling using local linear subspace structure can further enchance the dense NRSfM performance accuracy. The figure below show the visualization of such a linear subspace modeling using grassmannian representation.

Qualitative Results

Published Work and Other Useful Links

[1]. Organic Priors in Non-Rigid Structure from Motion, ECCV 2022.
[2]. Superpixel Soup: Monocular Dense 3D Reconstruction of a Complex Dynamic Scene, IEEE T-PAMI 2021.
[3]. Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited, IEEE/CVF WACV 2020.
[4]. Jumping manifolds: Geometry aware dense non-rigid structure from motion, IEEE/CVF CVPR 2019.
[5]. Scalable dense non-rigid structure-from-motion: A grassmannian perspective, IEEE/CVF CVPR 2018.
[6]. Monocular dense 3d reconstruction of a complex dynamic scene from two perspective frames, IEEE/CVF ICCV 2017.
[7]. Spatial-Temporal Union of Subspaces for Multi body Non-rigid Structure from Motion, Elsevier Pattern Recognition 2017.
[8]. Multi body Non-rigid Structure from Motion, IEEE 3DV 2016.

description arXiv page insert_comment BibTeX integration_instructions Dataset integration_instructions Code videocam Video

Project Accomplishment


		Recipient of Best Algorithm Award from Disney Research at NRSFM challenge CVPR 2017, Hawaii USA.
      		Nominated for J.G Crawford Prize for Best Doctoral Thesis 2019, ANU Canberra.
      		Recipient of HDR Merit Scholarship, funded in part by Australian Research Council.
      		Recipient of Vice-Chancellor Grant for CVPR 2018 Conference, Salt Lake City, Utah USA.

Authors and Acknowledgements