Non-Rigid Structure from Motion

Suryansh Kumar, ETH Zurich


The problem of recovering the 3D shape of a non-rigidly deforming object from its image feature correspondences across multiple frames is widely known as Non-Rigid Structure from Motion (NRSfM). It is a well-defined classical problem whose solution can assist several industrial applications in virtual reality, medical surgery, movies, etc. However, the task is challenging due to the inherent unconstrained nature of the problem itself, as many 3D varying configurations can have similar image projections. To date, no algorithm can solve NRSfM for all kinds of conceivable motion. Consequently, additional constraints, priors, and assumptions are often employed to solve NRSfM. Our work takes on this challenging problem and proposes algorithms that have set a new performance benchmark to solve NRSfM. Our solutions discuss the classical work in NRSfM and suggest necessary elementary modifications to it. The foundation of our contributions surpasses the traditional single object NRSFM and, for the first time, provides an effective formulation to realize multi-body NRSfM. Further, most NRSfM factorization methods can effectively handle only sparse feature correspondences. Later, sparse 3d points are used to approximate the object's dense shape via the organization of 3d points, planes, or other elementary geometric primitive. Nevertheless, sparse representation provides incomplete information about the object's global shape. On the contrary, our proposed methods can directly solve dense NRSfM showing state-of-the-art accuracy.

A visual illustration of basic pipeline setup for NRSfM factorization. The above dataset is taken from walking sequence introduced by Torresani et al. T-PAMI, 2008.

For dense NRSfM, we show that we can recover deforming shape 3d with favorable accuracy using just matrix factorization approach. However, a carefully modeling using local linear subspace structure can further enchance the dense NRSfM performance accuracy. The figure below show the visualization of such a linear subspace modeling using grassmannian representation.

We represent shape as a union of low-dimensional linear subspace.

Qualitative Results

Sparse NRSfM: Qualitative Results on CVPR NRSfM Challenge 2017 Dataset (Jensen et al. IJCV 2021).

Multi-body Sparse NRSfM: Qualitative Results on Motion Capture Dataset. We synthesized multi-body dataset using Akther et al. NIPS 2009 and Torresani et al. TPAMI 2008 dataset sequence.

Dense NRSfM: Qualitative Results on Garg et al. CVPR 2013, Varol et al. CVPR 2012 dataset.

Published Work and Other Useful Links

Project Accomplishment

  • Recipient of Best Algorithm Award from Disney Research at NRSFM challenge CVPR 2017, Hawaii USA.
  • Nominated for J.G Crawford Prize for Best Doctoral Thesis 2019, ANU Canberra.
  • Recipient of HDR Merit Scholarship, funded in part by Australian Research Council.
  • Recipient of Vice-Chancellor Grant for CVPR 2018 Conference, Salt Lake City, Utah USA.
  • Authors and Acknowledgements