[Paper Review] Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detection

Xin Feng, Yifeng Xu+2 more|arXiv (Cornell University)|Aug 27, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes a hierarchical contrastive learning framework for pattern-generalizable image corruption detection, detecting corrupted regions by learning semantic contrasts between corrupted and uncorrupted regions rather than relying on semantic patterns. The method uses coarse-to-fine mask prediction with a specialized hierarchical interaction mechanism, achieving state-of-the-art performance and strong generalization across diverse corruption patterns like graffiti, noise, and watermarks.

ABSTRACT

Effective image restoration with large-size corruptions, such as blind image inpainting, entails precise detection of corruption region masks which remains extremely challenging due to diverse shapes and patterns of corruptions. In this work, we present a novel method for automatic corruption detection, which allows for blind corruption restoration without known corruption masks. Specifically, we develop a hierarchical contrastive learning framework to detect corrupted regions by capturing the intrinsic semantic distinctions between corrupted and uncorrupted regions. In particular, our model detects the corrupted mask in a coarse-to-fine manner by first predicting a coarse mask by contrastive learning in low-resolution feature space and then refines the uncertain area of the mask by high-resolution contrastive learning. A specialized hierarchical interaction mechanism is designed to facilitate the knowledge propagation of contrastive learning in different scales, boosting the modeling performance substantially. The detected multi-scale corruption masks are then leveraged to guide the corruption restoration. Detecting corrupted regions by learning the contrastive distinctions rather than the semantic patterns of corruptions, our model has well generalization ability across different corruption patterns. Extensive experiments demonstrate following merits of our model: 1) the superior performance over other methods on both corruption detection and various image restoration tasks including blind inpainting and watermark removal, and 2) strong generalization across different corruption patterns such as graffiti, random noise or other image content. Codes and trained weights are available at https://github.com/xyfJASON/HCL .

Motivation & Objective

• To address the challenge of precise, pattern-generalizable corruption mask detection in large-scale image corruptions such as blind inpainting and watermark removal.
• To overcome the limitations of existing methods like VCNet, which rely on learning semantic patterns of corruptions and thus generalize poorly to unseen or irregular corruption patterns.
• To develop a general-purpose blind image restoration framework that detects corruption masks automatically and guides high-quality restoration without prior knowledge of corruption shapes or appearances.
• To improve performance and efficiency by leveraging coarse-to-fine contrastive learning with inter-scale knowledge propagation through a novel hierarchical interaction mechanism.

Proposed method

• The method employs a hierarchical contrastive learning framework that first predicts a coarse corruption mask using low-resolution features via contrastive learning.
• It then refines uncertain regions in the coarse mask using high-resolution contrastive learning on fine-grained features, focusing only on low-confidence pixels for efficiency.
• A specialized hierarchical interaction mechanism enables inter-stage semantic consistency and high-quality sample selection, promoting knowledge transfer between coarse and fine stages.
• The framework learns an embedding space where corrupted and uncorrupted regions are contrastively separated, avoiding reliance on semantic pattern recognition.
• The detected multi-scale masks guide a coarse-to-fine generative restoration process, improving restoration quality.
• The model is trained end-to-end using contrastive loss to maximize semantic distinction between corrupted and uncorrupted regions across scales.

Experimental results

Research questions

• RQ1 Can a contrastive learning-based approach generalize better than pattern-recognition-based methods like VCNet across diverse and unseen corruption patterns?
• RQ2 How effective is a coarse-to-fine hierarchical contrastive learning framework in detecting corruption masks with high precision and efficiency?
• RQ3 What is the contribution of the proposed hierarchical interaction mechanism in enhancing knowledge transfer and performance across scales?
• RQ4 To what extent does the model generalize to unseen corruption types such as graffiti, random noise, or real image patches not seen during training?

Key findings

• The proposed method achieves state-of-the-art performance on corruption detection, with a mean F1 score of 0.986 and IoU of 0.975 on the generalization benchmark, significantly outperforming VCNet.
• On blind image inpainting, the model achieves a PSNR of 37.69 and SSIM of 0.985 on graffiti corruption, surpassing VCNet (PSNR: 32.66, SSIM: 0.963) and matching or exceeding specialized methods.
• In watermark removal, the model achieves a PSNR of 41.88, SSIM of 0.992, and FID of 0.007, outperforming the state-of-the-art Split then Refine method (FID: 0.011).
• Ablation studies confirm that the hierarchical interaction mechanism, especially inter-stage semantic consistency, is critical—removing it reduces F1 by 0.002 and IoU by 0.015.
• The model generalizes strongly to unseen corruption patterns: on unseen graffiti and noise, it maintains high detection accuracy (F1 > 0.97) while VCNet’s performance drops significantly.
• The method reduces FID by 50% compared to VCNet on watermark removal and by 25% on shadow removal, demonstrating robustness across diverse restoration tasks.