[Paper Review] Hierarchical Fine-Grained Image Forgery Detection and Localization

Xiao Guo, Xiaohong Liu+4 more|arXiv (Cornell University)|Mar 30, 2023

Digital Media Forensic Detection4 citations

TL;DR

This paper proposes a hierarchical fine-grained framework for unified image forgery detection and localization (IFDL) that models forgery attributes across multiple levels of abstraction—ranging from general categories like 'fully synthesized' to specific methods like 'DDPM' or 'splicing'. By leveraging hierarchical dependencies between forgery attributes, the method improves both image-level detection and pixel-level localization, achieving state-of-the-art performance on seven benchmarks and generalizing to unseen forgery methods.

ABSTRACT

Differences in forgery attributes of images generated in CNN-synthesized and image-editing domains are large, and such differences make a unified image forgery detection and localization (IFDL) challenging. To this end, we present a hierarchical fine-grained formulation for IFDL representation learning. Specifically, we first represent forgery attributes of a manipulated image with multiple labels at different levels. Then we perform fine-grained classification at these levels using the hierarchical dependency between them. As a result, the algorithm is encouraged to learn both comprehensive features and inherent hierarchical nature of different forgery attributes, thereby improving the IFDL representation. Our proposed IFDL framework contains three components: multi-branch feature extractor, localization and classification modules. Each branch of the feature extractor learns to classify forgery attributes at one level, while localization and classification modules segment the pixel-level forgery region and detect image-level forgery, respectively. Lastly, we construct a hierarchical fine-grained dataset to facilitate our study. We demonstrate the effectiveness of our method on $7$ different benchmarks, for both tasks of IFDL and forgery attribute classification. Our source code and dataset can be found: \href{https://github.com/CHELSEA234/HiFi_IFDL}{github.com/CHELSEA234/HiFi-IFDL}.

Motivation & Objective

• To address the challenge of unified image forgery detection and localization across diverse forgery domains, including CNN-synthesized and image-editing methods.
• To model the inherent hierarchical structure of forgery attributes, such as moving from 'fully synthesized' to specific methods like 'diffusion models' or 'splicing'.
• To improve representation learning by jointly optimizing multi-branch feature extraction, hierarchical classification, and pixel-level localization.
• To construct a new hierarchical fine-grained dataset (HiFi-IFDL) to support training and evaluation of unified IFDL models.

Proposed method

• The framework uses a multi-branch feature extractor, where each branch learns features for a specific level of forgery attribute classification.
• Hierarchical classification is performed by conditioning each node’s prediction on the path from the root, ensuring consistency across levels.
• Pixel-level forgery localization is achieved via a contrastive loss that compares features to a mean feature vector from real images.
• The loss function is adaptive: when the input is real, only the final branch is trained; when forged, all branches are optimized.
• The model is trained end-to-end using separate learning rates for feature extractor, classification, and localization modules.
• A novel dataset, HiFi-IFDL, is constructed with 12 forgery categories spanning 4 levels of granularity, including both seen and unseen methods.

Experimental results

Research questions

• RQ1 Can a unified IFDL framework effectively detect and localize forgeries across both CNN-synthesized and image-editing domains?
• RQ2 How does modeling forgery attributes hierarchically improve detection and localization performance compared to flat classification?
• RQ3 To what extent can the model generalize to unseen forgery methods not present in the training set?
• RQ4 How does the hierarchical dependency between forgery attributes enhance feature representation learning?
• RQ5 What is the impact of multi-level supervision on localization accuracy and robustness?

Key findings

• The proposed method achieves state-of-the-art performance on seven benchmarks for both image-level forgery detection and pixel-level localization.
• The model generalizes effectively to unseen forgery methods, demonstrating robustness in zero-shot settings.
• Fine-grained classification at deeper levels (e.g., specific diffusion models) benefits from coarser-level supervision, improving accuracy even when coarse-level predictions are weak.
• The hierarchical structure enables better localization by leveraging the correlation between forgery method and forgery region distribution.
• The proposed HiFi-IFDL dataset provides a diverse, multi-level benchmark for future research in unified IFDL.
• The method shows strong robustness in qualitative results, correctly classifying complex forgeries such as those from StyleGANv3 and Faceshifter.