University of Oulu

P. Nguyen, A. Karnewar, L. Huynh, E. Rahtu, J. Matas and J. Heikkila, "RGBD-Net: Predicting Color and Depth Images for Novel Views Synthesis," 2021 International Conference on 3D Vision (3DV), 2021, pp. 1095-1105, doi: 10.1109/3DV53792.2021.00117

RGBD-Net : predicting color and depth images for novel views synthesis

Saved in:
Author: Nguyen, Phong1; Karnewar, Animesh2; Huynh, Lam1;
Organizations: 1University of Oulu
3Tampere University
4Czech Technical University in Prague
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 10.8 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2022-03-21


We propose a new cascaded architecture for novel view synthesis, called RGBD-Net, which consists of two core components: a hierarchical depth regression network and a depth-aware generator network. The former one predicts depth maps of the target views by using adaptive depth scaling, while the latter one leverages the predicted depths and renders spatially and temporally consistent target images. In the experimental evaluation on standard datasets, RGBD-Net not only outperforms the state-of-the-art by a clear margin, but it also generalizes well to new scenes without per-scene optimization. Moreover, we show that RGBD-Net can be optionally trained without depth supervision while still retaining high-quality rendering. Thanks to the depth regression network, RGBD-Net can be also used for creating dense 3D point clouds that are more accurate than those produced by some state-of-the-art multi-view stereo methods.

see all

Series: International Conference on 3D Vision proceedings
ISSN: 2378-3826
ISSN-E: 2475-7888
ISSN-L: 2378-3826
ISBN: 978-1-6654-2688-6
ISBN Print: 978-1-6654-2689-3
Pages: 1095 - 1105
DOI: 10.1109/3DV53792.2021.00117
Host publication: 2021 International Conference on 3D Vision (3DV)
Conference: International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.