[Paper Review] LSMI-Sinkhorn: Semi-supervised Squared-Loss Mutual Information Estimation with Optimal Transport.

Yanbin Liu, Makoto Yamada+4 more|arXiv (Cornell University)|Sep 5, 2019

Neural Networks and Applications25 references4 citations

TL;DR

This paper proposes LSMI-Sinkhorn, a semi-supervised squared-loss mutual information estimation method that leverages both limited paired samples and abundant unpaired data via optimal transport and quadratic programming. By formulating SMI through density ratio estimation and solving it with a Sinkhorn-regularized optimal transport approach, the method achieves accurate mutual information estimation even with few paired samples, demonstrating strong performance in image matching and photo album summarization.

ABSTRACT

Estimating mutual information is an important machine learning and statistics problem. To estimate the mutual information from data, a common practice is preparing a set of paired samples. However, in some cases, it is difficult to obtain a large number of data pairs. To address this problem, we propose squared-loss mutual information (SMI) estimation using a small number of paired samples and the available unpaired ones. We first represent SMI through the density ratio function, where the expectation is approximated by the samples from marginals and its assignment parameters. The objective is formulated using the optimal transport problem and quadratic programming. Then, we introduce the least-square mutual information-Sinkhorn algorithm (LSMI-Sinkhorn) for efficient optimization. Through experiments, we first demonstrate that the proposed method can estimate the SMI without a large number of paired samples. We also evaluate and show the effectiveness of the proposed LSMI-Sinkhorn on various types of machine learning problems such as image matching and photo album summarization.

Motivation & Objective

• To address the challenge of mutual information estimation when only a small number of paired samples are available.
• To improve SMI estimation accuracy by incorporating unpaired data alongside limited paired samples.
• To develop an efficient optimization framework for SMI estimation using optimal transport and quadratic programming.
• To evaluate the method on real-world machine learning tasks such as image matching and photo album summarization.

Proposed method

• The method represents squared-loss mutual information (SMI) through a density ratio function between joint and product-of-marginals distributions.
• It formulates the SMI estimation as an optimal transport problem with assignment parameters to link paired and unpaired samples.
• The objective function is solved via quadratic programming with Sinkhorn regularization to ensure computational efficiency and stability.
• The least-square mutual information-Sinkhorn (LSMI-Sinkhorn) algorithm optimizes the SMI objective using a differentiable transport plan derived from the Sinkhorn iteration.
• The method leverages unpaired samples to improve the estimation of the density ratio, reducing reliance on large paired datasets.
• The approach is designed to be scalable and effective even when paired data is scarce.

Experimental results

Research questions

• RQ1 Can SMI estimation be effectively improved using only a small number of paired samples and a large number of unpaired samples?
• RQ2 How does incorporating unpaired data through optimal transport enhance mutual information estimation accuracy?
• RQ3 What is the performance of the LSMI-Sinkhorn method in semi-supervised learning scenarios with limited paired data?
• RQ4 How does the method compare to existing SMI estimation techniques in image matching and photo album summarization tasks?

Key findings

• The proposed LSMI-Sinkhorn method achieves accurate SMI estimation even with a small number of paired samples, reducing dependency on large labeled datasets.
• The method demonstrates improved performance in image matching tasks by effectively utilizing unpaired data through optimal transport.
• In photo album summarization, LSMI-Sinkhorn outperforms baseline methods by leveraging mutual information estimation with limited paired examples.
• The Sinkhorn-regularized optimal transport formulation enables stable and efficient optimization, crucial for real-world applications.
• The experimental results confirm that the integration of unpaired data significantly enhances the robustness and accuracy of SMI estimation.