[Paper Review] Image Inpainting via Conditional Texture and Structure Dual Generation

Xiefan Guo, Hongyu Yang+1 more|arXiv (Cornell University)|Aug 22, 2021

Generative Adversarial Networks and Image Synthesis39 references18 citations

TL;DR

This paper proposes a two-stream generative network for image inpainting that jointly models structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner, using a Bi-directional Gated Feature Fusion (Bi-GFF) module and a Contextual Feature Aggregation (CFA) module to enhance consistency and detail. The method achieves state-of-the-art performance on CelebA, Paris StreetView, and Places2 datasets.

ABSTRACT

Deep generative approaches have recently made considerable progress in image inpainting by introducing structure priors. Due to the lack of proper interaction with image texture during structure reconstruction, however, current solutions are incompetent in handling the cases with large corruptions, and they generally suffer from distorted results. In this paper, we propose a novel two-stream network for image inpainting, which models the structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner so that they better leverage each other for more plausible generation. Furthermore, to enhance the global consistency, a Bi-directional Gated Feature Fusion (Bi-GFF) module is designed to exchange and combine the structure and texture information and a Contextual Feature Aggregation (CFA) module is developed to refine the generated contents by region affinity learning and multi-scale feature aggregation. Qualitative and quantitative experiments on the CelebA, Paris StreetView and Places2 datasets demonstrate the superiority of the proposed method. Our code is available at https://github.com/Xiefan-Guo/CTSDG.

Motivation & Objective

• To address the limitation of current deep generative methods in handling large corruptions due to weak interaction between structure and texture modeling.
• To improve global consistency and detail quality in image inpainting by explicitly coupling structure and texture generation.
• To develop a dual-stream architecture that enables mutual supervision between texture synthesis and structure reconstruction for more plausible results.
• To enhance feature consistency and long-range dependency through novel modules: Bi-GFF and CFA.

Proposed method

• The method employs a two-stream generator where one stream performs structure-constrained texture synthesis and the other performs texture-guided structure reconstruction, enabling bidirectional feature interaction.
• A Bi-directional Gated Feature Fusion (Bi-GFF) module is introduced to exchange and refine features between the structure and texture streams, improving consistency.
• A Contextual Feature Aggregation (CFA) module models long-range spatial dependencies through region affinity learning and multi-scale feature aggregation to enhance local detail fidelity.
• The discriminator is also dual-branch, with one branch assessing texture realism and the other evaluating structural sharpness, enabling joint optimization.
• The network is trained with adversarial loss, perceptual loss, and L1 loss to balance realism, structure, and pixel-level accuracy.
• The architecture is evaluated on CelebA, Paris StreetView, and Places2, with ablation studies on Bi-GFF, CFA, and multi-scale aggregation.

Experimental results

Research questions

• RQ1 Can jointly modeling texture synthesis and structure reconstruction in a dual-stream framework improve image inpainting quality compared to single-stream or sequential methods?
• RQ2 How does the Bi-directional Gated Feature Fusion (Bi-GFF) module affect the consistency between generated textures and structures?
• RQ3 To what extent does the Contextual Feature Aggregation (CFA) module enhance long-range spatial coherence and detail quality in inpainted regions?
• RQ4 Does multi-scale feature aggregation in CFA lead to measurable improvements in quantitative metrics and visual quality?
• RQ5 How does the proposed method compare to state-of-the-art approaches in terms of both qualitative and quantitative performance on standard benchmarks?

Key findings

• The proposed method achieves the new state-of-the-art on CelebA, Paris StreetView, and Places2 datasets, with a Fréchet Inception Distance (FID) of 0.039, 0.107, and 0.226, respectively.
• Ablation studies show that removing the Bi-GFF module increases FID to 0.045, 0.114, and 0.236, indicating its critical role in feature consistency.
• Eliminating the CFA module results in a FID increase to 0.049, 0.119, and 0.243, demonstrating its importance in preserving global structure and texture quality.
• The multi-scale feature aggregation in CFA improves FID by 0.004–0.008 compared to the original single-scale version, confirming its effectiveness.
• User studies with 10 expert participants show that the proposed method is significantly preferred over state-of-the-art methods in perceptual realism.
• The two-stream architecture with dual generation outperforms both single-stream baselines and multi-task single-stream networks, confirming the benefit of decoupled yet interactive modeling of structure and texture.