Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

Yanqin Jiang1, Li Zhang3, Jin Gao1, Weiming Hu1, Yao Yao2*
1 CASIA, 2 Nanjin University, 3 Fudan University
Paper arXiv arXiv Video (coming soon) Code Data

(a) In-the-wild Video

(b) Foreground

(c) Consistent4D

(d) Consistent4D Depth

Consistent4D generates a 360° dynamic object from a uncalibrated monocular video
captured by hand-held devices like smartphones or from segments of animated films.

Abstract

In this paper, we present Consistent4D, a novel approach for generating 4D dynamic objects from uncalibrated monocular videos.

Uniquely, we cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model as the primary supervision signal for training Dynamic Neural Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to facilitate stable convergence and temporal continuity under the supervision signal which is discrete along the time axis. To achieve spatial and temporal consistency, we further introduce an Interpolation-driven Consistency Loss. It is optimized by minimizing the discrepancy between rendered frames from DyNeRF and interpolated frames from a pretrained video interpolation model.

Extensive experiments show that our Consistent4D can perform competitively to prior art alternatives, opening up new possibilities for 4D dynamic object generation from monocular videos, whilst also demonstrating advantage for conventional text-to-3D generation tasks.

Pipeline
Pipeline Image
The framework consists of the optimization of a Cascade DyNeRF and the training of a post video enhancer. The Cascade DyNeRF, which adopts residual learning strategy, is supervised by SDS loss from an image-to-image diffusion model. Particularly, a novel Interpolation-driven Consistency Loss is proposed to compensate for the spatiotemporal inconsistency brought by the image diffusion model. For post-processing, we train a lightweight cross-frame video enhancer using GAN to further improve the quality of the video rendered from DyNeRF.

Gallery

Input Video
Novel View 0
Novel View 1
Timestamp 0
Timestamp 1
Please refer to the paper and the video to see more visualization results.
BibTeX

@inproceedings{
jiang2024consistentd,
title={Consistent4D: Consistent 360{\textdegree} Dynamic Object Generation from Monocular Video},
author={Yanqin Jiang and Li Zhang and Jin Gao and Weiming Hu and Yao Yao},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sPUrdFGepF}
}