[Paper Review] Object Discovery with a Copy-Pasting GAN

Relja Arandjelović, Andrew Zisserman|arXiv (Cornell University)|May 27, 2019

Advanced Neural Network Applications65 references42 citations

TL;DR

Unsupervised object discovery by training a copy-pasting GAN where a generator learns to copy object regions from a source image into a destination image to fool a discriminator, with mechanisms to prevent shortcuts and stabilize training. The approach shows strong object discovery performance on cluttered datasets and competitive results against generative baselines.

ABSTRACT

We tackle the problem of object discovery, where objects are segmented for a given input image, and the system is trained without using any direct supervision whatsoever. A novel copy-pasting GAN framework is proposed, where the generator learns to discover an object in one image by compositing it into another image such that the discriminator cannot tell that the resulting image is fake. After carefully addressing subtle issues, such as preventing the generator from `cheating', this game results in the generator learning to select objects, as copy-pasting objects is most likely to fool the discriminator. The system is shown to work well on four very different datasets, including large object appearance variations in challenging cluttered backgrounds.

Motivation & Objective

• Motivate unsupervised learning of object discovery without direct supervision.
• Propose a copy-pasting GAN (CP-GAN) where the generator outputs a copy-mask to blend images.
• Ensure training stability by preventing generator shortcuts and stabilizing the discriminator.
• Demonstrate object discovery performance on diverse, cluttered datasets and analyze configurations.

Proposed method

• Generator outputs a copy-mask m_theta(I_s) from a source image I_s to blend with a destination I_d (I_c = m_theta(I_s) * I_s + (1 - m_theta(I_s)) * I_d).
• Discriminator learns to distinguish real vs. composite images, guiding the generator to copy coherent objects.
• To prevent trivial shortcuts, introduce an anti-shortcut loss using an irrelevant image I_i to penalize universally applicable masks.
• Blur discriminator input to avoid low-level cue cheating, and inject grounded fake images to keep the discriminator learning.
• Provide auxiliary mask-prediction and reinforcement-learning-inspired seed selection (Instance Colouring) for multi-object scenarios.
• Use two CP-GAN generator architectures: Direct (single-channel mask) and Instance Colouring (dense feature map plus seed for RL-based mask).

Experimental results

Research questions

• RQ1 Can an unsupervised CP-GAN discover and segment objects by learning to copy objects between images?
• RQ2 What mechanisms are necessary to prevent generator shortcuts and stabilize GAN training in this setting?
• RQ3 How does the CP-GAN perform on varied synthetic datasets with cluttered backgrounds and appearance variation?
• RQ4 How do different generator architectures affect object discovery performance?
• RQ5 Can the method scale to more complex, real-world-like images while remaining unsupervised?

Key findings

• CP-GAN achieves high object discovery rates on Squares (98.3%), NoisySquares (100%), and CLEVR+bg (98.3%).
• On Flying Chairs (64x64 with clutter) CP-GAN reaches 40.3% object discovery probability.
• Instance Colouring generator architecture generally outperforms Direct, especially on CLEVR+bg (98.3% vs 61.6%).
• Ablation shows anti-shortcut + border-zeroing are critical for avoiding trivial solutions; auxiliary mask prediction and grounded fakes improve stability and accuracy.
• Blurred discriminator inputs improve training stability, particularly for CLEVR+bg; grounding fakes and mask-prediction guide the discriminator.
• CP-GAN generally outperforms a purely generative baseline MONet on cluttered scenes and shows resilience to background distractors.