[논문 리뷰] StructureFlow: Image Inpainting via Structure-aware Appearance Flow

Yurui Ren, Xiaoming Yu외 4명|arXiv (Cornell University)|2019. 08. 11.

Generative Adversarial Networks and Image Synthesis참고 문헌 35인용 수 56

한 줄 요약

StructureFlow는 이미지 인페인팅을 구조 재구성과 질감 생성으로 분할하고, 전역 구조를 위해 에지 보존된 매끄러운 이미지를 사용하며, 재구성된 구조를 가이드로 하는 질감 합성을 위해 Gaussian sampling이 적용된 appearance flow를 활용한다.

ABSTRACT

Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.

연구 동기 및 목표

• Improve global structure recovery in inpainting by using edge-preserved smooth images as structure guidance.
• Enable vivid texture synthesis by sampling features from regions with similar structures.
• Stabilize appearance flow training with Gaussian sampling and a sampling correctness loss.
• Demonstrate effectiveness on diverse datasets and analyze ablations to validate components.

제안 방법

• Two-stage architecture with a structure reconstructor Gs and a texture generator Gt.
• Structure reconstructor uses edge-preserved smooth images Se as guidance and minimizes L1 loss to Sgt with adversarial training.
• Texture generator uses appearance flow to warp and sample features from existing regions based on reconstructed structures, aided by a sampling correctness loss.
• Gaussian sampling replaces bilinear sampling in appearance flow to expand receptive field.
• Training includes L1, adversarial, and sampling-correctness loss terms with stage-wise pretraining and subsequent fine-tuning.

실험 결과

연구 질문

• RQ1 Can edge-preserved smooth structures effectively guide global structure reconstruction for inpainting?
• RQ2 Does appearance flow guided texture generation with Gaussian sampling produce more realistic textures and coherent structures?
• RQ3 How do ablations of structure guidance, flow sampling, and loss terms affect inpainting quality on diverse datasets?

주요 결과

• StructureFlow achieves competitive PSNR, SSIM, and FID on Places2 compared to CA, EdgeConnect, and PConv.
• Subjective user studies show StructureFlow outperforming competitors in highly structured scenes (Celeba and Paris) and remaining competitive on Places2.
• Ablation shows including the structure reconstructor improves performance; too little or too much smoothing (sigma) degrades results; appearance flow with Gaussian sampling and sampling-correctness loss yields better texture realism and flow stability.
• Gaussian sampling expands the receptive field of appearance flow, mitigating poor gradient propagation and enabling long-range feature matching.
• The sampling-correctness loss, based on VGG feature cosine similarities, helps constrain sampling to semantically relevant regions.