University of Oulu

K. Songsri-in and S. Zafeiriou, "Face Video Generation from a Single Image and Landmarks," 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020, pp. 69-76, doi: 10.1109/FG47880.2020.00104

Face video generation from a single image and landmarks

Saved in:
Author: Songsri-in, Kritaphat1; Zafeiriou, Stefanos1,2
Organizations: 1Imperial College London
2University of Oulu
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2.4 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe2022032124239
Language: English
Published: IEEE, 2020
Publish Date: 2022-03-21
Description:

Abstract

In this paper, we are concerned with the challenging problem of producing a full image sequence of a deformable face given only an image and generic facial motions encoded by a set of sparse landmarks. To this end, we build upon recent breakthroughs in image-to-image translation such as pix2pix, CycleGAN and StarGAN which learn Deep Convolutional Neural Networks (DCNNs) that learn to map aligned pairs of images between different domains (i.e., having different labels) and propose a new architecture which is not driven any more by labels but by spatial maps, facial landmarks. In particular, we propose the MotionGAN which transforms an input face image into a new one according to a heatmap of target landmarks. We show that it is possible to create very realistic face videos using a single image and a set of target landmarks. Furthermore, our method can be used to edit a facial image with arbitrary motions according to landmarks (e.g., expression, speech, etc.). This provides much more flexibility to face editing, expression transfer, facial video creation, etc. than models based on discrete expressions, audio or action units.

see all

ISBN: 978-1-7281-3079-8
ISBN Print: 978-1-7281-3080-4
Pages: 69 - 76
DOI: 10.1109/FG47880.2020.00104
OADOI: https://oadoi.org/10.1109/FG47880.2020.00104
Host publication: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)
Conference: IEEE International Conference on Automatic Face and Gesture Recognition and Workshops
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Subjects:
Copyright information: © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.