[Paper Review] Coarse-to-fine Seam Estimation for Image Stitching

Tianli Liao, Jing Chen+1 more|arXiv (Cornell University)|May 24, 2018

Advanced Image and Video Retrieval Techniques16 references3 citations

TL;DR

This paper proposes a coarse-to-fine seam estimation method for image stitching that improves perceptual quality by iteratively refining seams using a patch-point evaluation algorithm. By reweighting the difference map based on pixel correlation and variance along the seam, the method achieves perception-consistent results superior to conventional seam-cutting and existing seam-driven approaches.

ABSTRACT

Seam-cutting and seam-driven techniques have been proven effective for handling imperfect image series in image stitching. Generally, seam-driven is to utilize seam-cutting to find a best seam from one or finite alignment hypotheses based on a predefined seam quality metric. However, the quality metrics in most methods are defined to measure the average performance of the pixels on the seam without considering the relevance and variance among them. This may cause that the seam with the minimal measure is not optimal (perception-inconsistent) in human perception. In this paper, we propose a novel coarse-to-fine seam estimation method which applies the evaluation in a different way. For pixels on the seam, we develop a patch-point evaluation algorithm concentrating more on the correlation and variation of them. The evaluations are then used to recalculate the difference map of the overlapping region and reestimate a stitching seam. This evaluation-reestimation procedure iterates until the current seam changes negligibly comparing with the previous seams. Experiments show that our proposed method can finally find a nearly perception-consistent seam after several iterations, which outperforms the conventional seam-cutting and other seam-driven methods.

Motivation & Objective

• Address the limitation of conventional seam-cutting methods that rely on average-based quality metrics, which may not align with human perception.
• Overcome the issue where the seam with the lowest metric score is not necessarily the most visually consistent or optimal.
• Develop a method that refines stitching seams by focusing on local correlation and variance of pixels along the seam.
• Achieve perception-consistent stitching results by iteratively re-estimating the seam using refined difference maps.
• Provide a robust, iterative framework that converges to a nearly perception-consistent seam without requiring multiple alignment hypotheses.

Proposed method

• Initiate seam estimation using conventional seam-cutting with a standard difference map based on color intensity differences.
• Apply a patch-point evaluation algorithm to assess pixels along the current seam, focusing on local correlation and variance.
• Use the evaluation scores to modify the difference map: increase smoothness costs for pixels with high evaluation values (indicating potential misalignment).
• Recompute the difference map in a banding region (5 pixels on each side of the seam) using an exponential weighting function: $ f(x) = e^{ au(x - ho)} $, where $ x $ is the evaluation score.
• Re-estimate the stitching seam via graph cuts using the updated energy function based on the recalculated difference map.
• Iterate the evaluation and re-estimation process until the new seam is contained within the previous banding area, indicating convergence.

Experimental results

Research questions

• RQ1 Can a seam with minimal average quality metric still be perceptually suboptimal due to high variance or poor local correlation?
• RQ2 Can iterative refinement based on local pixel correlation and variance improve the perceptual quality of stitched images?
• RQ3 Does reweighting the difference map using patch-point evaluation lead to a more consistent and visually plausible seam?
• RQ4 Can a coarse-to-fine strategy that refines the seam through multiple iterations achieve better results than single-pass seam-cutting?
• RQ5 How does the proposed method compare to existing seam-driven approaches in terms of perceptual consistency and visual artifact reduction?

Key findings

• The proposed method produces stitching results that are more consistent with human perception than conventional seam-cutting and other seam-driven methods.
• The iterative refinement process converges within 5 iterations on average, achieving stable and perceptually optimal seams.
• Visual artifacts such as structural inconsistencies (e.g., misaligned edges or textures) are significantly reduced, especially in complex or parallax-prone regions.
• The patch-point evaluation effectively identifies misaligned pixels, as evidenced by high evaluation scores in regions with visual inconsistencies.
• The method outperforms SEAGULL and Parallax on public datasets like Parallax and SEAGULL, particularly in preserving structural coherence.
• The final stitching result, generated via gradient domain fusion, exhibits smoother transitions and fewer visible seams compared to baseline methods.