[Paper Review] Deep Video Inpainting Detection

Peng Zhou, Ning Yu+4 more|arXiv (Cornell University)|Jan 26, 2021

Generative Adversarial Networks and Image Synthesis44 references9 citations

TL;DR

This paper proposes VIDNet, a novel end-to-end deep learning framework for detecting inpainted regions in videos by leveraging multimodal features from RGB and Error Level Analysis (ELA) frames, combined with a quad-directional local attention module and ConvLSTM-based temporal modeling. VIDNet achieves state-of-the-art performance in both in-domain and cross-domain video inpainting detection, outperforming existing methods with strong generalization and robustness to compression and noise perturbations.

ABSTRACT

This paper studies video inpainting detection, which localizes an inpainted region in a video both spatially and temporally. In particular, we introduce VIDNet, Video Inpainting Detection Network, which contains a two-stream encoder-decoder architecture with attention module. To reveal artifacts encoded in compression, VIDNet additionally takes in Error Level Analysis frames to augment RGB frames, producing multimodal features at different levels with an encoder. Exploring spatial and temporal relationships, these features are further decoded by a Convolutional LSTM to predict masks of inpainted regions. In addition, when detecting whether a pixel is inpainted or not, we present a quad-directional local attention module that borrows information from its surrounding pixels from four directions. Extensive experiments are conducted to validate our approach. We demonstrate, among other things, that VIDNet not only outperforms by clear margins alternative inpainting detection methods but also generalizes well on novel videos that are unseen during training.

Motivation & Objective

• To address the critical need for detecting malicious video inpainting, which can enable misinformation and evidence tampering.
• To develop the first learning-based framework for video inpainting detection, as prior work has largely focused on image-level manipulation detection.
• To improve detection robustness by leveraging multimodal cues from RGB and ELA features to reveal compression artifacts.
• To model spatial and temporal dependencies explicitly through a quad-directional local attention module and ConvLSTM decoder for accurate spatiotemporal localization of inpainted regions.

Proposed method

• VIDNet employs a two-stream encoder-decoder architecture that processes both RGB and ELA frames to extract multimodal features at multiple scales.
• The encoder uses a pretrained VGG network to extract features from RGB frames, while ELA frames are used to reveal compression inconsistencies and artifacts.
• A quad-directional local attention (QDLA) module is applied to the final RGB feature map to attend to neighboring pixels from four directions, enhancing spatial context modeling.
• Multimodal features from different scales are fused and fed into a four-layer Convolutional LSTM decoder that models temporal dynamics across video frames.
• The decoder uses skip connections from coarser layers to preserve spatial details and improve mask prediction accuracy.
• The entire network is trained end-to-end with backpropagation using binary cross-entropy loss for pixel-wise inpainting mask prediction.

Experimental results

Research questions

• RQ1 Can a deep learning framework effectively detect inpainted regions in videos by combining RGB and ELA features?
• RQ2 How does modeling spatial context through quad-directional attention improve detection performance compared to standard attention or global pooling?
• RQ3 To what extent can a video inpainting detection model generalize to unseen inpainting methods and datasets not seen during training?
• RQ4 How robust is the proposed method under common video distortions such as Gaussian noise and JPEG compression?
• RQ5 Does incorporating temporal modeling via ConvLSTM lead to more temporally consistent and accurate inpainting mask predictions?

Key findings

• VIDNet achieves a mean IoU of 0.257 and F1 score of 0.367 on the Free-form Video Inpainting (FVI) dataset, significantly outperforming the next best method (GSR-Net) by 70% in F1 score.
• On the DAVIS 2016 dataset, VIDNet outperforms all competing methods with clear margins, demonstrating superior detection accuracy and robustness.
• The model generalizes well to out-of-domain inpainted videos, maintaining strong performance even when tested on videos generated with different inpainting methods than those in the training data.
• Adding noise and JPEG augmentation during training improves robustness to perturbations, with VIDNet maintaining high performance under SNR 20 dB and JPEG quality 70.
• The ablation study confirms that combining RGB and ELA features, along with the QDLA module and ConvLSTM, leads to the highest performance, validating the contribution of each component.
• Qualitative results show that VIDNet produces more temporally consistent and spatially accurate masks than frame-by-frame methods like GSR-Net, which suffer from flickering and inconsistency.