[Paper Review] TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations

Yuqian Zhou, Connelly Barnes+2 more|arXiv (Cornell University)|Mar 29, 2021

Generative Adversarial Networks and Image Synthesis70 references4 citations

TL;DR

TransFill proposes a reference-guided image inpainting method that uses multiple homographies to align a source image to a target image with a hole, applying learned color and spatial transformations for consistency, then fusing results via a learned mask. It achieves state-of-the-art performance on complex holes with parallax, lighting differences, and large deformations by combining depth-based homography estimation, bilateral color transformation, and multi-proposal fusion.

ABSTRACT

Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image. Most existing technologies exploit patch similarities within the image, or leverage large-scale training data to fill the hole using learned semantic and texture information. However, due to the ill-posed nature of the inpainting task, such methods struggle to complete larger holes containing complicated scenes. In this paper, we propose TransFill, a multi-homography transformed fusion method to fill the hole by referring to another source image that shares scene contents with the target image. We first align the source image to the target image by estimating multiple homographies guided by different depth levels. We then learn to adjust the color and apply a pixel-level warping to each homography-warped source image to make it more consistent with the target. Finally, a pixel-level fusion module is learned to selectively merge the different proposals. Our method achieves state-of-the-art performance on pairs of images across a variety of wide baselines and color differences, and generalizes to user-provided image pairs.

Motivation & Objective

• To address the challenge of filling large, complex holes in images where self-similarity or single-image generation fails.
• To enable robust image inpainting when a reference image of the same scene is available, even with significant viewpoint changes and lighting differences.
• To overcome limitations of global homography by modeling parallax through multiple depth-estimated homographies.
• To improve visual consistency between source and target images through end-to-end learned color and spatial transformation.
• To achieve high-quality inpainting in regions with missing pixels due to occlusion or warping by fusing multiple transformed proposals and applying single-image inpainting as a backup.

Proposed method

• Estimate feature correspondences between target and source images, then cluster inliers by depth to infer multiple depth planes.
• For each depth cluster, compute a homography to register the source image to the target, generating multiple initial alignment proposals.
• Apply a deep bilateral color transformation to harmonize color distribution between source and target images for each proposal.
• Learn a per-pixel spatial transformation (warp) to refine alignment and correct residual geometric distortions.
• Train a fusion network to learn per-pixel weights that combine multiple transformed proposals into a single coherent output.
• Integrate a pre-trained single-image inpainting model (ProFill) to complete regions missing after warping, with fusion weights learned to combine this with the multi-proposal results.

Figure 2: System pipeline. Given the target image $I_{t}^{M}$ masked by an associated binary hole image $M$ , and a single source image $I_{s}$ , we first propose multiple global homographies using the multi-homography proposal module, and locally adjust color and spatial misalignments in each propo

Experimental results

Research questions

• RQ1 Can multiple homographies, each aligned to different depth planes, improve image inpainting performance on large holes with parallax compared to a single global homography?
• RQ2 How effective is a learned color and spatial transformation in aligning source images with different lighting and exposure to the target image?
• RQ3 Does fusing multiple transformed source proposals using a learnable fusion mask yield better results than single or unweighted fusion?
• RQ4 How does the proposed method generalize to user-provided image pairs with large viewpoint differences and complex scene layouts?
• RQ5 To what extent does the integration of a single-image inpainting model improve results in regions where source image content is missing after warping?

Key findings

• TransFill achieves a PSNR of 39.59, SSIM of 0.9919, and LPIPS of 0.0116 on RealEstate10K with a frame displacement of 10, outperforming prior state-of-the-art methods.
• With a frame displacement of 30, TransFill maintains a PSNR of 35.62 and LPIPS of 0.0213, demonstrating robustness to large geometric shifts.
• The depth-based clustering of feature points for homography estimation outperforms random and spatial clustering, achieving 37.576 PSNR when using five homographies.
• The color-spatial transformation module is critical: removing either color or spatial transformation reduces PSNR by over 1.0, showing their complementary role.
• The full pipeline with both color-spatial transformation and fusion network (CST + SPF) achieves the best performance (37.576 PSNR), significantly outperforming ablations without these components.
• In user studies, TransFill achieved a 95.65% preference rate over baselines, with p < 10^-6, indicating strong user preference for its results.

Figure 3: Multi-homography Proposal Module. We compute the monocular depth $D_{t}$ of the non-hole region $I_{t}^{M}$ , and cluster the feature matching points into $N$ sub-groups using the depth values. Each estimated homography $H_{i}$ will align different regions within the hole. ${}^{*}H_{6}$ in