[Paper Review] Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network

Nasim Souly, Concetto Spampinato+1 more|arXiv (Cornell University)|Mar 28, 2017

Generative Adversarial Networks and Image Synthesis25 references53 citations

TL;DR

The paper develops semi- and weakly supervised semantic segmentation using a Generative Adversarial Network where a generator creates fake images to regularize a multiclass pixel classifier (discriminator). Conditioning with weak labels further improves generated sample quality and segmentation performance across multiple datasets.

ABSTRACT

Semantic segmentation has been a long standing challenging task in computer vision. It aims at assigning a label to each image pixel and needs significant number of pixellevel annotated data, which is often unavailable. To address this lack, in this paper, we leverage, on one hand, massive amount of available unlabeled or weakly labeled data, and on the other hand, non-real images created through Generative Adversarial Networks. In particular, we propose a semi-supervised framework ,based on Generative Adversarial Networks (GANs), which consists of a generator network to provide extra training examples to a multi-class classifier, acting as discriminator in the GAN framework, that assigns sample a label y from the K possible classes or marks it as a fake sample (extra class). The underlying idea is that adding large fake visual data forces real samples to be close in the feature space, enabling a bottom-up clustering process, which, in turn, improves multiclass pixel classification. To ensure higher quality of generated images for GANs with consequent improved pixel classification, we extend the above framework by adding weakly annotated data, i.e., we provide class level information to the generator. We tested our approaches on several challenging benchmarking visual datasets, i.e. PASCAL, SiftFLow, Stanford and CamVid, achieving competitive performance also compared to state-of-the-art semantic segmentation method

Motivation & Objective

• Leverage unlabeled data to regularize deep segmentation models under a semi-supervised setting.
• Explore the use of GANs where the discriminator performs pixel-level multiclass classification for segmentation.
• Enhance GAN training with weak supervision by conditioning on image-level labels to improve sample quality and segmentation accuracy.
• Evaluate the proposed framework on multiple challenging datasets and compare with state-of-the-art methods.

Proposed method

• Replace the standard GAN discriminator with a fully convolutional multiclass pixel classifier producing K+1 class confidences (K real classes plus a fake class).
• Train the generator to produce images that push the discriminator to assign real class labels to pixels, while also producing realistic samples that resemble training data.
• Use three data streams for the discriminator: labeled data, unlabeled data, and generated (fake) data, with a pixel-wise cross-entropy loss and a fake-data term.
• Formulate the semi-supervised loss for the discriminator to maximize log(D(x)) for real data, minimize cross-entropy for labeled data, and minimize log(1−D(G(z))) for fake data.
• Provide a weakly supervised extension via conditional GANs where image-level labels l are used to condition both G and D, enforcing label presence in generated images and guiding the discriminator with image-level supervision.
• In weak supervision, concatenate one-hot class vectors to the generator input to produce class-conditioned images and adapt the loss to include image-level label consistency.

Experimental results

Research questions

• RQ1 Can unlabeled data and synthetic samples from a GAN improve pixel-level semantic segmentation performance compared to fully supervised training?
• RQ2 Does conditioning the GAN on weak image-level labels further improve generated sample quality and downstream segmentation performance?
• RQ3 How do semi- and weakly supervised GAN-based frameworks perform across standard benchmarks (VOC 2012, SiftFlow, StanfordBG, CamVid) relative to state-of-the-art methods?

Key findings

• On VOC 2012 val, semi-supervised training yields pixel acc 90.5, mean acc 80.7, mean IU 64.1, outperforming the fully supervised baseline of 89.9/69.2/59.5.
• On VOC 2012 val with 30% fully labeled data, semi-supervised improves mean IU to 42.2 and pixel acc to 83.6, while weakly supervised reaches 44.6 mean IU and 84.6 pixel acc.
• On VOC 2012 val, weakly supervised surpasses several state-of-the-art semi-supervised methods in mean IU and mean accuracy, with 65.8 mean IU and 91.3 pixel acc.
• On SiftFlow, semi-supervised improves mean IU to 35.1 and pixel acc to 86.3 over the fully supervised baseline of 83.4/34.4, respectively.
• Across StanfordBG and CamVid, semi-supervised and weakly supervised GANs outperform fully supervised baselines and competing methods, with notable gains in mean accuracy and mean IU.
• Qualitative results show generated images capture dataset-specific structures (e.g., roads at bottom, sky at top) that aid the discriminator in learning meaningful pixel-level features.