[Paper Review] W-Net: A Deep Model for Fully Unsupervised Image Segmentation

Xide Xia, Brian Kulis|arXiv (Cornell University)|Nov 22, 2017

Advanced Neural Network ApplicationsComputer Science20 references214 citations

TL;DR

W-Net combines two U-Net–style FCNs into an encoder-decoder for unsupervised image segmentation, trained with a soft normalized cut loss and reconstruction loss, followed by CRF smoothing and hierarchical merging. It achieves competitive BSDS performance without labeled data.

ABSTRACT

While significant attention has been recently focused on designing supervised deep semantic segmentation algorithms for vision tasks, there are many domains in which sufficient supervised pixel-level labels are difficult to obtain. In this paper, we revisit the problem of purely unsupervised image segmentation and propose a novel deep architecture for this problem. We borrow recent ideas from supervised semantic segmentation methods, in particular by concatenating two fully convolutional networks together into an autoencoder--one for encoding and one for decoding. The encoding layer produces a k-way pixelwise prediction, and both the reconstruction error of the autoencoder as well as the normalized cut produced by the encoder are jointly minimized during training. When combined with suitable postprocessing involving conditional random field smoothing and hierarchical segmentation, our resulting algorithm achieves impressive results on the benchmark Berkeley Segmentation Data Set, outperforming a number of competing methods.

Motivation & Objective

• Motivate unsupervised image segmentation in domains with scarce pixelwise labels.
• Propose a deep encoder-decoder architecture (W-Net) that yields a dense, k-way pixel-wise segmentation without supervision.
• Jointly optimize a soft normalized cut loss on the encoder output and a reconstruction loss of the autoencoder.
• Enhance segmentation quality via postprocessing: fully connected CRF smoothing and hierarchical merging.

Proposed method

• Two FCNs in series form an autoencoder: an encoder (U_Enc) outputs a K-way pixel-wise soft segmentation and a decoder (U_Dec) reconstructs the input from the encoding.
• A differentiable soft-Ncut loss (J_soft-Ncut) is minimized on the encoder output to promote intra-segment affinity and inter-segment dissimilarity.
• A reconstruction loss (J_reconstr) minimizes the difference between input X and the reconstructed image from the encoding.
• Training alternates: update U_Enc to minimize J_soft-Ncut, then update both U_Enc and U_Dec to minimize J_reconstr.
• Depthwise separable convolutions are used across most modules to improve parameter efficiency; the network is fully convolutional (no fully connected layers).
• Postprocessing uses a fully connected CRF to sharpen boundaries and a hierarchical merging (UCM-based) to produce final segments.

Experimental results

Research questions

• RQ1 Can a fully unsupervised deep architecture produce competitive pixel-wise image segmentations without labeled data?
• RQ2 Does coupling an encoder-derived soft segmentation with a reconstruction objective yield segmentations that align with input structure?
• RQ3 What benefit do postprocessing steps (CRF smoothing and hierarchical merging) provide to fully unsupervised segmentations?
• RQ4 How does the proposed W-Net perform on standard unsupervised segmentation benchmarks like BSDS300/BSDS500 compared to classical methods?

Key findings

• W-Net achieves competitive scores on BSDS300/BSDS500 compared to classical unsupervised methods.
• Including soft-Ncut during training yields more coherent segmentations at the cost of some reconstruction quality.
• Postprocessing with CRF smoothing and hierarchical merging improves final segmentation quality (ODS/OIS) over the raw W-Net outputs.
• On BSDS benchmarks, W-Net approaches human-level performance on some metrics and surpasses several classical methods.
• Training without supervision uses datasets like PASCAL VOC2012 for representation learning, evaluated solely via BSDS ground truth.