University of Oulu

J. Cai, H. Han, J. Cui, J. Chen, L. Liu and S. K. Zhou, "Semi-Supervised Natural Face De-Occlusion," in IEEE Transactions on Information Forensics and Security, vol. 16, pp. 1044-1057, 2021, doi: 10.1109/TIFS.2020.3023793

Semi-supervised natural face de-occlusion

Saved in:
Author: Cai, Jiancheng1; Han, Hu1; Cui, Jiyun1;
Organizations: 1Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing, 100190, China
2Peking University Shenzhen Graduate School
3University of Oulu, Finland
4National University of Defense Technology, China
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 4.1 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2021-04-26


Occlusions are often present in face images in the wild, e.g., under video surveillance and forensic scenarios. Existing face de-occlusion methods are limited as they require the knowledge of an occlusion mask. To overcome this limitation, we propose in this paper a new generative adversarial network (named OA-GAN) for natural face de-occlusion without an occlusion mask, enabled by learning in a semi-supervised fashion using (i) paired images with known masks of artificial occlusions and (ii) natural images without occlusion masks. The generator of our approach first predicts an occlusion mask, which is used for filtering the feature maps of the input image as a semantic cue for de-occlusion. The filtered feature maps are then used for face completion to recover a non-occluded face image. The initial occlusion mask prediction might not be accurate enough, but it gradually converges to the accurate one because of the adversarial loss we use to perceive which regions in a face image need to be recovered. The discriminator of our approach consists of an adversarial loss, distinguishing the recovered face images from natural face images, and an attribute preserving loss, ensuring that the face image after de-occlusion can retain the attributes of the input face image. Experimental evaluations on the widely used CelebA dataset and a dataset with natural occlusions we collected show that the proposed approach can outperform the state of the art methods in natural face de-occlusion.

see all

Series: IEEE transactions on information forensics and security
ISSN: 1556-6013
ISSN-E: 1556-6021
ISSN-L: 1556-6013
Volume: 16
Pages: 1044 - 1057
DOI: 10.1109/TIFS.2020.3023793
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Funding: This work was supported in part by the National Key R&D Program of China (grant 2018AAA0102501), Natural Science Foundation of China (grant 61672496), and Youth Innovation Promotion Association CAS (2018135).
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.