[Paper Review] Leveraging Multi-view Image Sets for Unsupervised Intrinsic Image Decomposition and Highlight Separation

Renjiao Yi, Ping Tan+1 more|arXiv (Cornell University)|Nov 17, 2019

Image Enhancement Techniques30 references4 citations

TL;DR

This paper proposes an unsupervised deep learning method for intrinsic image decomposition and highlight separation using multi-view real images, such as customer product photos. It introduces a color distribution-based image representation that reduces sensitivity to misalignment and a contrastive loss that improves highlight and shading separation by leveraging their interdependence, achieving state-of-the-art performance on both tasks without ground-truth labels.

ABSTRACT

We present an unsupervised approach for factorizing object appearance into highlight, shading, and albedo layers, trained by multi-view real images. To do so, we construct a multi-view dataset by collecting numerous customer product photos online, which exhibit large illumination variations that make them suitable for training of reflectance separation and can facilitate object-level decomposition. The main contribution of our approach is a proposed image representation based on local color distributions that allows training to be insensitive to the local misalignments of multi-view images. In addition, we present a new guidance cue for unsupervised training that exploits synergy between highlight separation and intrinsic image decomposition. Over a broad range of objects, our technique is shown to yield state-of-the-art results for both of these tasks.

Motivation & Objective

• To develop an unsupervised method for decomposing object images into albedo, shading, and highlight layers using real-world multi-view image sets.
• To address the challenge of training on misaligned multi-view images from sources like customer product photos, which are common but difficult to align precisely.
• To improve the reliability of intrinsic image decomposition by modeling the interdependence between highlight separation and shading estimation.
• To achieve state-of-the-art performance on both highlight separation and intrinsic image decomposition without requiring ground-truth annotations.

Proposed method

• The method uses a multi-view dataset of customer product photos with large illumination variations to train the network in an unsupervised manner.
• It introduces a novel image representation based on local color distributions that discard fine-scale positional information, making the training robust to local misalignments.
• A color distribution loss is proposed to enforce consistency across multi-view images by comparing local color histograms, reducing sensitivity to misalignment.
• A contrastive loss is introduced between shading estimates computed with and without the highlight sub-network, guiding the network to produce more accurate highlight and shading components.
• The network is trained end-to-end using a combination of unsupervised losses and finetuned on real data after pretraining on a small fraction of synthetic ShapeNet data.
• The architecture jointly optimizes for highlight, shading, and albedo layers using the image formation model $I_d = A \cdot S$.

Experimental results

Research questions

• RQ1 Can multi-view real images from unstructured sources like customer product photos be effectively used to train unsupervised intrinsic image decomposition and highlight separation?
• RQ2 How can training be made robust to local misalignments in multi-view image sets that are common in real-world data?
• RQ3 Can the relationship between highlight separation and intrinsic decomposition be exploited to improve both tasks simultaneously?
• RQ4 What is the performance gain of using a color distribution-based representation over standard pixel-wise losses in the presence of misalignment?
• RQ5 Can a contrastive loss that leverages the dependency between highlight and shading estimation improve the overall decomposition quality?

Key findings

• On the DiLiGenT dataset, the proposed method achieved the lowest shading MSE (0.0041) and DSSIM (0.0316), outperforming SIRFS, DI, Shi et al., Li et al., and CG.
• On the ShapeNet Intrinsic Dataset, the method achieved state-of-the-art performance in both MSE and DSSIM for intrinsic decomposition, with significant improvements from unsupervised finetuning and the color distribution loss.
• Ablation studies showed that removing the contrastive loss led to degenerate solutions where the diffuse layer became all zero, confirming its critical role in network stability.
• The color distribution loss led to a 5–48% improvement in performance over pixel-wise low-rank losses, especially for shading estimation, which better captured surface shape.
• Despite being finetuned on only 1.1% of the ShapeNet dataset, the method generalized well to synthetic images and outperformed methods trained on larger synthetic splits.
• Qualitative results on real images showed that the end-to-end system produced sharper albedo maps with better texture details compared to prior methods.