[Paper Review] Bridging Global Context Interactions for High-Fidelity Image Completion

Chuanxia Zheng, Tat‐Jen Cham+2 more|arXiv (Cornell University)|Apr 2, 2021

Generative Adversarial Networks and Image Synthesis54 references4 citations

TL;DR

This paper proposes TFill, a transformer-based image completion method that treats image completion as a sequence-to-sequence task to explicitly model long-range global context without local bias. By using a restrictive CNN for non-overlapping, small-receptive-field token representation, the model ensures equal attention to distant visible regions, while an attention-aware layer (AAL) improves appearance consistency, achieving state-of-the-art results on multiple benchmarks with lower FID and LPIPS scores.

ABSTRACT

Bridging global context interactions correctly is important for high-fidelity image completion with large masks. Previous methods attempting this via deep or large receptive field (RF) convolutions cannot escape from the dominance of nearby interactions, which may be inferior. In this paper, we propose to treat image completion as a directionless sequence-to-sequence prediction task, and deploy a transformer to directly capture long-range dependence in the encoder. Crucially, we employ a restrictive CNN with small and non-overlapping RF for weighted token representation, which allows the transformer to explicitly model the long-range visible context relations with equal importance in all layers, without implicitly confounding neighboring tokens when larger RFs are used. To improve appearance consistency between visible and generated regions, a novel attention-aware layer (AAL) is introduced to better exploit distantly related high-frequency features. Overall, extensive experiments demonstrate superior performance compared to state-of-the-art methods on several datasets.

Motivation & Objective

• To address the challenge of long-range context modeling in high-fidelity image completion with large, irregular masks.
• To overcome the local bias in deep convolutional networks that favor nearby over distant visible regions.
• To enable explicit, equal-weighted modeling of global dependencies across all visible tokens in the transformer encoder.
• To improve appearance consistency between generated and visible regions through adaptive feature selection.
• To design a flexible, fully convolutional framework that scales to arbitrary image resolutions.

Proposed method

• Treats image completion as a directionless sequence-to-sequence prediction task using a transformer encoder.
• Employs a restrictive CNN with small, non-overlapping receptive fields to generate token representations, isolating visible context from local correlation.
• Uses self-attention in the transformer to model long-range dependencies equally across all visible tokens, avoiding proximity dominance.
• Introduces an attention-aware layer (AAL) that dynamically selects high-frequency features based on attention scores, improving appearance consistency.
• Adopts a two-stage refinement framework: coarse content generation via TFill- Coarse, followed by appearance refinement using AAL.
• Uses a fully convolutional design to enable inference on arbitrary image sizes, overcoming the fixed sequence length limitation of standard position embeddings.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture effectively model long-range global context in image completion without being biased toward local features?
• RQ2 How does restricting the receptive field of the token encoder affect the modeling of global dependencies in image completion?
• RQ3 Can an attention-aware layer improve appearance consistency by adaptively selecting features from both visible and generated regions?
• RQ4 How does the proposed method compare to existing two-stage approaches in terms of fidelity and generalization across diverse mask types?
• RQ5 Can the model scale to high-resolution images while maintaining performance and efficiency?

Key findings

• TFill achieves state-of-the-art performance on multiple datasets, reducing FID by 2.8% and LPIPS by 6.0% on average compared to prior methods.
• The restrictive CNN tokenization reduces FID to 3.63 and LPIPS to 0.057 when using a 16×16 receptive field, outperforming larger RFs.
• The AAL module reduces FID by 0.11 and LPIPS by 0.004 compared to the next best baseline, demonstrating improved appearance consistency.
• On the FFHQ dataset, TFill- AAL achieves LPIPS of 0.0412 and FID of 2.57 on random masks, outperforming SA, CA, and SLTA.
• The model maintains low memory (1.15 GB) and inference time (0.180 s) even with large receptive fields, showing efficiency.
• Qualitative results show no artifacts in generated regions, unlike methods such as CA and PIC, which produce color-inconsistent outputs.