Novel View Synthesis with Pixel-Space Diffusion Models - Apple Machine Learning Research
research area Computer Vision conference CVPR content type paper published February 2025 Novel View Synthesis with Pixel-Space Diffusion Models Authors Noam Elata, Bahjat Kawar, Yaron Ostrovsky-Berman, Miriam Farber, Ron Sokolovsky View publication View source code (GitHub) Synthesizing a novel view from a single input image is a challenging task. Traditionally, this task was approached by estimating scene depth, warping, and inpainting, with machine learning models enabling parts of the pipeline.
More recently, generative models are being increasingly employed in novel view synthesis (NVS), often encompassing the entire end-to-end system. In this work, we adapt a modern diffusion model architecture for end-to-end NVS in the pixel space, substantially outperforming previous state-of-the-art (SOTA) techniques. We explore different ways to encode geometric information into the network. Our experiments show that while these methods may enhance performance, their impact is minor compared to utilizing improved generative models.
Moreover, we introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance compared to their multi-view counterparts. This leads to improved generalization capabilities to scenes with out-of-domain content. Figure 1. Novel view synthesis results from our diffusion model. Source views are taken from RealEstate10K, and fed into our base and SR models to produce a 256x256-pixel prediction.
Our end-to-end system implicitly learns to preserve the features in the source view, transform their position along with the camera movement, and generate realistic details in unseen areas. Related readings and updates. Pseudo-Generalized Dynamic View Synthesis from a Video May 2, 2024 research area Computer Vision conference ICLR Rendering scenes observed in a monocular video from novel viewpoints is a chal- lenging problem.
For static scenes the community has studied both scene-specific optimization techniques, which optimize on every test scene, and generalized tech- niques, which only run a deep net forward pass on a test scene. In contrast, for dy- namic scenes, scene-specific optimization techniques exist, but, to our best knowl- edge, there is currently no… Read more Fast and Explicit Neural View Synthesis February 2, 2022 research area Computer Vision , research area Methods and Algorithms conference WACV We study the problem of novel view synthesis from sparse source observations of a scene comprised of 3D objects.
We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. Our approach explicitly encodes observations into a volumetric representation that enables amortized rendering. We demonstrate that although continuous radiance field representations have gained a lot of… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day.
Work with us