PREF: Predictability Regularized Neural Motion Fields

ECCV 2022 (Oral Presentation)

Liangchen Song^1,2, Xuan Gong^1,2, Benjamin Planche², Meng Zheng²,

David Doermann¹, Junsong Yuan¹, Terrence Chen², and Ziyan Wu²

¹ University at Buffalo, Buffalo NY, USA

² United Imaging Intelligence, Cambridge MA, USA

Paper

Video Presentation

Video presentation of proposed PREF method.

Abstract

Knowing the 3D motions in a dynamic scene is essential to many vision applications. Recent progress is mainly focused on estimating the activity of some specific elements like humans. In this paper, we leverage a neural motion field for estimating the motion of all points in a multiview setting. Modeling the motion from a dynamic scene with multiview data is challenging due to the ambiguities in points of similar color and points with time-varying color. We propose to regularize the estimated motion to be predictable. If the motion from previous frames is known, then the motion in the near future should be predictable. Therefore, we introduce a predictability regularization by first conditioning the estimated motion on latent embeddings, then by adopting a predictor network to enforce predictability on the embeddings. The proposed framework PREF (Predictability REgularized Fields) achieves on par or better results than state-of-the-art neural motion field-based dynamic scene representation methods, while requiring no prior knowledge of the scene.

Overview of the proposed framework. Three networks are trained jointly: the space-time field, the motion field and the predictor. The space-time field returns color and occupancy for each point at a specific time. The motion field predicts the motion of a point based on a motion embedding vector. The predictor generates the future motion embedding based on previously observed embeddings.

Results

Qualitative results. Our method can handle topologically-varying scenes and estimate physical motion for all points in the space.

Citation



@InProceedings{uii_eccv22_pref,
  title={PREF: Predictability Regularized Neural Motion Fields},
  author={Song, Liangchen and Gong, Xuan and Planche, Benjamin and Zheng, Meng 
          and Doermann, David and Yuan, Junsong and Chen, Terrence and Wu, Ziyan},
  booktitle={European Conference on Computer Vision},
  year={2022},
}