University of Oulu

Nguyen-Ha P., Huynh L., Rahtu E., Heikkilä J. (2021) Sequential View Synthesis with Transformer. In: Ishikawa H., Liu CL., Pajdla T., Shi J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science, vol 12625. Springer, Cham.

Sequential view synthesis with transformer

Saved in:
Author: Nguyen-Ha, Phong1; Huynh, Lam1; Rahtu, Esa2;
Organizations: 1Center for Machine Vision and Signal Analysis, University of Oulu, Finland
2Computer Vision Group, Tampere University, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 10.8 MB)
Persistent link:
Language: English
Published: Springer Nature, 2021
Publish Date: 2021-05-12


This paper addresses the problem of novel view synthesis by means of neural rendering, where we are interested in predicting the novel view at an arbitrary camera pose based on a given set of input images from other viewpoints. Using the known query pose and input poses, we create an ordered set of observations that leads to the target view. Thus, the problem of single novel view synthesis is reformulated as a sequential view prediction task. In this paper, the proposed Transformer-based Generative Query Network (T-GQN) extends the neural-rendering methods by adding two new concepts. First, we use multi-view attention learning between context images to obtain multiple implicit scene representations. Second, we introduce a sequential rendering decoder to predict an image sequence, including the target view, based on the learned representations. Finally, we evaluate our model on various challenging datasets and demonstrate that our model not only gives consistent predictions but also doesn’t require any retraining for finetuning.

see all

Series: Lecture notes in computer science
ISSN: 0302-9743
ISSN-E: 1611-3349
ISSN-L: 0302-9743
ISBN: 978-3-030-69538-5
ISBN Print: 978-3-030-69537-8
Pages: 695 - 711
DOI: 10.1007/978-3-030-69538-5_42
Host publication: Computer Vision – ACCV 2020. ACCV 2020
Host publication editor: Ishikawa, H.
Liu, C.
Pajdla, T.
Shi, J.
Conference: Asian Conference on Computer Vision
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Copyright information: © Springer Nature Switzerland AG 2021. This is a post-peer-review, pre-copyedit version of an article published in Computer Vision – ACCV 2020. ACCV 2020. The final authenticated version is available online at: