[Paper Review] Unsupervised Eyeglasses Removal in the Wild

Bingwen Hu, Zhedong Zheng+3 more|arXiv (Cornell University)|Sep 16, 2019

Face recognition and analysis65 references4 citations

TL;DR

This paper proposes an unsupervised eyeglasses removal method using a unified generative adversarial network (ERGAN) that simultaneously learns to remove and generate eyeglasses, enabling robust performance across diverse eyeglass types and facial poses in the wild. The method achieves state-of-the-art realism and diversity in removal while improving downstream tasks like face verification and facial expression recognition.

ABSTRACT

Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses and sunglasses, and recovering appropriate eyes. Due to the large visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called Eyeglasses Removal Generative Adversarial Network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, i.e., removing and wearing eyeglasses. Given two facial images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a pre-processing method for these tasks.

Motivation & Objective

• Address the challenge of eyeglasses removal in unconstrained, real-world environments where facial poses, lighting, and eyeglass types vary significantly.
• Overcome the limitations of existing methods that require paired training data (same identity with/without glasses) and frontal face alignment.
• Develop a scalable, unified model that generalizes across diverse eyeglass types—rimless, full-rim, sunglasses—without dedicated systems per type.
• Enable the model to perform both eyeglasses removal and generation through a dual learning scheme, enhancing feature learning and realism.
• Demonstrate the method’s utility as a pre-processing tool for downstream face-related tasks such as face verification and facial expression recognition.

Proposed method

• Propose a unified Generative Adversarial Network (ERGAN) that jointly learns eyeglasses removal and generation via a dual learning framework.
• Use weakly supervised learning: only binary labels (with or without glasses) are required, eliminating the need for precise bounding boxes or paired images.
• Train the model on 202,599 face images from CelebA, split into two sets: with glasses and without glasses, enabling large-scale, weakly annotated learning.
• Implement a rotation-based pre-processing step to align faces around the face center, avoiding complex landmark-based alignment and improving robustness to pose variations.
• Employ a cycle-consistent GAN architecture that enforces identity preservation and realistic eye region generation by swapping the eye region between two images (with and without glasses).
• Focus the generator and discriminator on the eye region to improve local detail fidelity and reduce the complexity of full-face generation.

Experimental results

Research questions

• RQ1 Can a unified, unsupervised GAN-based method effectively remove diverse eyeglass types in unconstrained, real-world face images without requiring paired training data?
• RQ2 How does jointly learning eyeglasses removal and generation improve the quality and realism of the generated results compared to single-task approaches?
• RQ3 To what extent does the proposed method improve downstream face recognition tasks such as face verification and facial expression recognition?
• RQ4 Can the method generalize across varying facial poses and eyeglass appearances without requiring explicit alignment or pose normalization?
• RQ5 Does the dual learning scheme enhance feature learning and identity preservation in the absence of paired supervision?

Key findings

• The proposed ERGAN method achieves competitive performance in eyeglasses removal in terms of realism and diversity, outperforming previous state-of-the-art methods.
• On the MeGlass dataset, face verification accuracy improved from 73.65% (with glasses) to 78.59% (after removal), approaching the 80.61% accuracy on images without glasses.
• The average Euclidean distance between matched face pairs decreased significantly after removal, indicating better preservation of identity and reduced occlusion interference.
• Facial expression recognition accuracy dropped by 6.9% when glasses were present (73.65% vs. 80.61% on non-glasses images), but improved to 78.59% after removal, showing strong benefit for expression recognition.
• Qualitative results show that images with glasses were frequently misclassified (e.g., as angry or neutral), but after removal, all were correctly classified as happy, demonstrating practical utility.
• The method is robust to pose variations due to rotation-based pre-processing, avoiding complex alignment and maintaining performance across diverse head poses.