[Paper Review] Universal Face Restoration With Memorized Modulation

Jia Li, Huaibo Huang+2 more|arXiv (Cornell University)|Oct 3, 2021

Advanced Image Processing Techniques48 references4 citations

TL;DR

This paper proposes a novel universal face restoration framework, Restoration with Memorized Modulation (RMM), which uses adaptive attentional modulation from noise and unsupervised wavelet memory to restore diverse low-quality face images. By integrating learnable noise priors, wavelet-based high-frequency texture memory, and multi-level normalization maps (AdaAO, AdaALN, AdaAIN), RMM achieves state-of-the-art performance on both synthetic and real-world degradation, demonstrating strong generalization across heterogeneous domains like cartoons, oil paintings, and NIR images.

ABSTRACT

Blind face restoration (BFR) is a challenging problem because of the uncertainty of the degradation patterns. This paper proposes a Restoration with Memorized Modulation (RMM) framework for universal BFR in diverse degraded scenes and heterogeneous domains. We apply random noise as well as unsupervised wavelet memory to adaptively modulate the face-enhancement generator, considering attentional denormalization in both layer and instance levels. Specifically, in the training stage, the low-level spatial feature embedding, the wavelet memory embedding obtained by wavelet transformation of the high-resolution image, as well as the disentangled high-level noise embeddings are integrated, with the guidance of attentional maps generated from layer normalization, instance normalization, and the original feature map. These three embeddings are respectively associated with the spatial content, high-frequency texture details, and a learnable universal prior against other blind image degradation patterns. We store the spatial feature of the low-resolution image and the corresponding wavelet style code as key and value in the memory unit, respectively. In the test stage, the wavelet memory value whose corresponding spatial key is the most matching with that of the inferred image is retrieved to modulate the generator. Moreover, the universal prior learned from the random noise has been memorized by training the modulation network. Experimental results show the superiority of the proposed method compared with the state-of-the-art methods, and a good generalization in the wild.

Motivation & Objective

• Address the challenge of blind face restoration (BFR) under diverse, unknown degradation patterns such as blur, noise, and compression.
• Overcome limitations of reference-based or pretrained-model-dependent methods by eliminating reliance on high-quality references or external models during inference.
• Develop a unified framework that generalizes across heterogeneous domains, including real-world, cartoon, and artistic face images.
• Introduce a memory-augmented mechanism that stores and retrieves high-frequency texture details from wavelet-transformed features for adaptive restoration.
• Improve restoration fidelity by jointly modeling spatial content, texture details, and a universal noise prior through attention-guided feature modulation.

Proposed method

• Propose the RMM module (RM³) that uses three adaptive attentional maps—AdaAO, AdaALN, and AdaAIN—derived from original, layer-normalized, and instance-normalized features to guide feature modulation.
• Integrate three embeddings: low-level spatial features (content), high-frequency wavelet coefficients (texture), and disentangled noise embeddings (universal prior) for multi-level restoration control.
• Implement a Wavelet Memory Module (WMM) that stores wavelet-style codes as values and low-resolution spatial features as keys, enabling retrieval of matching high-frequency details during inference.
• Use random noise with fixed latent size as a resolution-independent universal prior, trained end-to-end to modulate the generator via a learned modulation network.
• Apply adaptive feature modulation at multiple decoder stages using attention maps to dynamically balance structural integrity and texture detail synthesis.
• Train the model in an end-to-end manner using adversarial loss and perceptual losses, with memory retrieval performed during inference based on spatial feature similarity.

Experimental results

Research questions

• RQ1 Can a unified face restoration framework achieve superior performance across diverse, unseen degradation patterns without relying on high-quality reference images?
• RQ2 How effective is unsupervised wavelet memory in preserving high-frequency texture details during blind face restoration?
• RQ3 To what extent does a learnable, resolution-independent noise prior improve generalization in blind face restoration?
• RQ4 How do multi-level attentional maps (from original, layer, and instance normalization) contribute to balancing structural and texture restoration?
• RQ5 Can the proposed memorized modulation mechanism generalize to heterogeneous domains such as cartoons, pencil drawings, and NIR images?

Key findings

• RMM achieves the best FID (100.61), KID (8.24), and NIQE (4.54) scores among all variants on the FF-HQ dataset, outperforming models using wavelet decomposition from deeper VGG layers.
• The variant using wavelet packet decomposition of the input image (RMM) achieves the lowest NIQE (4.54) and best FID (100.61), indicating superior perceptual quality and fidelity.
• Using shallow-layer features (e.g., conv1_1) for wavelet decomposition yields the best NIQE (4.29) but worse FID (109.41) and KID (9.29), suggesting trade-offs between perceptual quality and structural accuracy.
• Query vectors from deeper layers (e.g., ResNet18 layers 0, 2, 4, 6) yield higher FID and KID scores (106.9–107.33) than RMM, indicating that shallow-layer queries are less effective for spatial feature matching.
• Ablation studies confirm that both noise modulation ($w/o ext{ }Z_N$) and wavelet memory ($w/o ext{ }Z_W$) significantly degrade performance, proving their essential role in training and inference.
• Failure cases occur in heterogeneous domains when cartoon faces resemble real faces (e.g., similar eye size), leading to loss of stylized features; extreme makeup also causes misalignment in facial component restoration.