[Paper Review] High-Quality Correspondence and Segmentation Estimation for Dual-Lens Smart-Phone Portraits

Xiaoyong Shen, Hongyun Gao+3 more|arXiv (Cornell University)|Apr 7, 2017

Advanced Image and Video Retrieval Techniques26 references3 citations

TL;DR

This paper proposes a joint optimization framework using a fully connected conditional random field (CRF) to simultaneously improve dense correspondence and semantic segmentation in dual-lens smartphone portrait images. By introducing regional correspondence to reduce computational cost and leveraging human body priors, the method achieves state-of-the-art performance, reducing average end point error to 5.29 and improving mean IoU to 88.33% on a new 2,000-image portrait dataset.

ABSTRACT

Estimating correspondence between two images and extracting the foreground object are two challenges in computer vision. With dual-lens smart phones, such as iPhone 7Plus and Huawei P9, coming into the market, two images of slightly different views provide us new information to unify the two topics. We propose a joint method to tackle them simultaneously via a joint fully connected conditional random field (CRF) framework. The regional correspondence is used to handle textureless regions in matching and make our CRF system computationally efficient. Our method is evaluated over 2,000 new image pairs, and produces promising results on challenging portrait images.

Motivation & Objective

• To address the challenge of accurate dense correspondence and semantic segmentation in dual-lens smartphone portrait images, which are complicated by textureless regions, occlusions, and noise.
• To unify correspondence estimation and semantic segmentation into a single joint optimization framework that improves both tasks through mutual refinement.
• To reduce computational cost in fully connected CRF inference by introducing regional correspondence, limiting the number of labels to under 40 per image.
• To improve performance on challenging portrait data, particularly around human body boundaries and in low-texture regions, using human body priors and robust feature matching.

Proposed method

• A joint fully connected CRF model is formulated to integrate correspondence and segmentation features, enabling mutual refinement between dense optical flow and per-pixel semantic labels.
• Regional correspondence is introduced to restrict the CRF label space, replacing per-pixel displacement maps with region-wise labels, reducing inference complexity from O(N²) to O(N) with N as the number of image regions.
• The method uses a coarse-to-fine optical flow initialization from existing models (e.g., MDP, LDOF) and refines it using CNN-based features and CRF optimization with human body priors.
• Semantic segmentation is initialized via fine-tuned FCN or CRFasRNN models on a new portrait-specific dataset and further refined using correspondence-guided CRF inference.
• A joint optimization scheme alternates between updating correspondence and segmentation using a unified CRF energy function that models spatial consistency and appearance similarity.
• The CRF energy function incorporates data terms for flow smoothness, segmentation confidence, and compatibility between correspondence and segmentation labels.

Experimental results

Research questions

• RQ1 Can joint optimization of correspondence and segmentation using a fully connected CRF improve performance on dual-lens portrait images compared to independent methods?
• RQ2 How effective is regional correspondence in reducing computational cost while maintaining or improving accuracy in dense correspondence estimation?
• RQ3 To what extent do human body priors and mutual refinement between correspondence and segmentation reduce errors in textureless and boundary regions?
• RQ4 Can a joint model outperform separate refinement of correspondence and segmentation in portrait images?

Key findings

• The proposed joint model achieves a mean intersection-over-union (IoU) of 88.33% on the new portrait dataset, significantly outperforming baseline models like FCN (79.51%), DeepLab (80.09%), and CRFasRNN (80.23%).
• The joint model reduces average end point error (AEPE) to 5.29, surpassing all compared methods including MDP (8.23), LDOF (8.32), and DeepFlow (7.87).
• Using regional correspondence reduces inference time from 186.3 seconds to 16.63 seconds per image while improving AEPE from 6.45 to 5.29, demonstrating both speed and accuracy gains.
• The separate refinement of segmentation only (Ours-separate) achieves 84.32% IoU, but the joint model improves this to 88.33%, proving the benefit of mutual refinement.
• Visual comparisons show that the method significantly reduces errors in human body boundaries and textureless regions, such as skin and hair, where prior methods fail.
• The method is robust to low-texture and occluded regions by leveraging high-level human body structure and correspondence consistency.