Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model

Created by
  • Haebom

Author

Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, Yueqi Duan

Outline

In this paper, we propose ReconX, a novel method for accurate 3D scene reconstruction from limited viewpoint images. Unlike existing dense-view based reconstruction methods, ReconX reconstructs the sparse-view reconstruction problem as a temporal generation task by leveraging the powerful generative prior knowledge of pre-trained video diffusion models. First, we generate a global point cloud from limited input viewpoints and encode it into a 3D structure condition. Based on this condition, the video diffusion model synthesizes video frames with high 3D consistency while preserving details. Finally, we reconstruct the 3D scene from the generated video using a confidence-based 3D Gaussian Splatting optimization technique. Experimental results show that ReconX outperforms existing state-of-the-art methods in terms of performance and generalization capability.

Takeaways, Limitations

Takeaways:
A novel approach to the problem of sparse viewpoint 3D scene reconstruction
Effectively leveraging the powerful generative ability of pre-trained video diffusion models for 3D reconstruction
Achieving improved reconstruction quality and generalization performance over existing methods
Efficient 3D scene restoration using a reliability-based 3D Gaussian Splatting optimization technique
Limitations:
May depend on the performance of pre-trained video diffusion models
The computational cost of generating 3D structure conditions and synthesizing video frames can be high.
Additional evaluation of generalization performance for different types of scenes may be needed.
Possible performance degradation under certain conditions (e.g. extremely rare times)
👍