[Paper Review] SeGAN: Segmenting and Generating the Invisible

Kiana Ehsani, Roozbeh Mottaghi+1 more|arXiv (Cornell University)|Mar 29, 2017

Advanced Neural Network Applications49 references4 citations

TL;DR

SeGAN proposes a GAN-based model that jointly segments and generates the invisible parts of occluded objects using synthetic, photo-realistic training data. It outperforms state-of-the-art methods in segmenting invisible regions, generates realistic appearances, and infers depth layering from occlusion relationships, generalizing effectively to natural images without fine-tuning.

ABSTRACT

Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental results show that: (a) SeGAN can learn to generate the appearance of the occluded parts of objects; (b) SeGAN outperforms state-of-the-art segmentation baselines for the invisible parts of objects; (c) trained on synthetic photo realistic images, SeGAN can reliably segment natural images; (d) by reasoning about occluder occludee relations, our method can infer depth layering.

Motivation & Objective

• To address the challenge of inferring the appearance and segmentation of occluded object parts in scenes where objects are partially hidden.
• To jointly optimize for segmentation of invisible regions and generation of their realistic appearance, overcoming limitations of separate segmentation or generation approaches.
• To learn from large-scale, accurate synthetic data with precise occlusion boundaries, avoiding the subjectivity and noise of human-annotated invisible region data.
• To enable zero-shot generalization from synthetic training data to real-world natural images for unseen object instances.
• To infer depth layering by modeling occluder-occludee relationships, enabling relative depth ordering without explicit depth supervision.

Proposed method

• SeGAN uses a conditional GAN framework where the generator predicts both the segmentation mask and appearance of invisible object parts given an input image and visible mask.
• The model employs a two-stream generator: one branch predicts the invisible segmentation mask (SI), and the other generates the pixel-level appearance for those regions.
• The discriminator is trained to distinguish between real and generated image-mask pairs, enforcing realism in both appearance and spatial consistency.
• Training is performed exclusively on synthetic, photo-realistic scenes with ground-truth occlusion boundaries, enabling precise supervision for invisible regions.
• The architecture is category-agnostic and does not require semantic category labels, relying instead on spatial and occlusion reasoning.
• Depth layering is inferred by checking intersection-over-union between visible masks of potential occluders and predicted invisible masks of occludees, with a 5% threshold.

Experimental results

Research questions

• RQ1 Can a deep learning model jointly learn to segment and generate the appearance of invisible object parts in occluded scenes?
• RQ2 Does joint optimization of segmentation and generation lead to better performance than separate or sequential approaches?
• RQ3 Can a model trained on synthetic data generalize to real-world natural images without fine-tuning?
• RQ4 Can the model infer depth layering from occlusion patterns, even without explicit depth supervision?
• RQ5 How does the model’s performance compare to state-of-the-art baselines in segmentation, generation, and depth ordering?

Key findings

• SeGAN outperforms state-of-the-art segmentation baselines in predicting invisible regions, achieving a SI IoU of 58.9% on synthetic data.
• On natural images, SeGAN generalizes effectively, achieving a SI IoU of 50.7% despite being trained only on synthetic data.
• The model generates realistic appearances for invisible parts, outperforming GAN-based baselines in appearance generation quality.
• SeGAN achieves 84.04% accuracy in depth layering inference on synthetic data, significantly outperforming the single-image depth baseline (60.18%).
• Combining synthetic and natural data during training degrades performance, suggesting domain shift challenges.
• The model successfully infers occluder-occludee relationships by analyzing mask intersections, enabling relative depth ordering from occlusion cues alone.