[Paper Review] A DCA-Like Algorithm and its Accelerated Version with Application in Data Visualization

Hoai An Le Thi, Hoai Minh Le+2 more|arXiv (Cornell University)|Jun 25, 2018

Neural Networks and Applications14 references3 citations

TL;DR

This paper proposes DCA-Like and Accelerated DCA-Like, two novel variants of the Difference of Convex functions Algorithm (DCA) for solving constrained sum-of-differentiable-and-composite-functions minimization problems. By iteratively updating the Lipschitz constant parameter μ to improve majorization, DCA-Like accelerates convergence without requiring an accurate estimate of L, while Accelerated DCA-Like further enhances speed via Nesterov’s acceleration. The methods achieve up to 9.2× faster convergence than standard DCA and outperform it in objective value on 4 out of 6 benchmark datasets, including t-SNE visualization tasks.

ABSTRACT

In this paper, we present two variants of DCA (Different of Convex functions Algorithm) to solve the constrained sum of differentiable function and composite functions minimization problem, with the aim of increasing the convergence speed of DCA. In the first variant, DCA-Like, we introduce a new technique to iteratively modify the decomposition of the objective function. This successive decomposition could lead to a better majorization and consequently a better convergence speed than the basic DCA. We then incorporate the Nesterov's acceleration technique into DCA-Like to give rise to the second variant, named Accelerated DCA-Like. The convergence properties and the convergence rate under Kudyka-Lojasiewicz assumption of both variants are rigorously studied. As an application, we investigate our algorithms for the t-distributed stochastic neighbor embedding. Numerical experiments on several benchmark datasets illustrate the efficiency of our algorithms.

Motivation & Objective

• To address the slow convergence of standard DCA when the Lipschitz constant L of the objective function is unknown or poorly estimated.
• To develop a DCA variant that dynamically updates the parameter μ to improve majorization and accelerate convergence without requiring L to be known in advance.
• To incorporate Nesterov’s acceleration technique into the DCA-Like framework to further speed up convergence.
• To rigorously analyze the convergence properties and convergence rates of both variants under the Kurdyka-Łojasiewicz (KŁ) assumption.
• To evaluate the proposed algorithms on t-distributed Stochastic Neighbor Embedding (t-SNE) for data visualization, demonstrating practical efficiency and effectiveness.

Proposed method

• DCA-Like introduces an iterative μ-updating strategy that modifies the decomposition of the objective function to achieve better majorization, even when μ is smaller than the true Lipschitz constant L.
• The method ensures convergence to a critical point despite the lack of a valid DC decomposition at each step, relying on the structure of the problem and convergence guarantees under KŁ.
• Accelerated DCA-Like integrates Nesterov’s momentum-based extrapolation into DCA-Like to accelerate convergence, improving the descent behavior of the iterates.
• The algorithms are applied to t-SNE by reformulating the optimization problem as a DC program, where the convex subproblems can be solved explicitly.
• Convergence analysis is conducted under the Kurdyka-Łojasiewicz (KŁ) inequality, which ensures global convergence of bounded sequences and provides convergence rate estimates.
• The proposed algorithms are compared with standard DCA and the state-of-the-art Majorization-Minimization (MM) algorithm for t-SNE, which is shown to be equivalent to DCA-Like in this context.

Experimental results

Research questions

• RQ1 Can a DCA variant be designed to accelerate convergence without requiring an accurate estimate of the Lipschitz constant L?
• RQ2 Does iteratively updating the parameter μ in the majorization step lead to better convergence speed and improved objective values?
• RQ3 Can Nesterov’s acceleration technique be successfully integrated into a DCA framework to further enhance convergence speed?
• RQ4 Under what conditions does the proposed DCA-Like and Accelerated DCA-Like converge globally, and what convergence rates can be established?
• RQ5 How do DCA-Like and Accelerated DCA-Like perform in practice on real-world data visualization tasks such as t-SNE compared to existing state-of-the-art methods?

Key findings

• DCA-Like reduces the number of iterations by 3 to 11.5 times compared to standard DCA, resulting in a 1.9 to 9.2 times improvement in computing time across six benchmark datasets.
• DCA-Like achieves a lower objective value than standard DCA on 4 out of 6 datasets (letters, shuttle, miniboone, covertype), with comparable performance on the remaining two.
• Accelerated DCA-Like reduces the number of iterations by 1.8 to 5 times compared to DCA-Like, achieving a 1.5 to 5 times speedup in computation time.
• ADCA-Like achieves the best objective value on 5 out of 6 datasets, outperforming both DCA and DCA-Like in solution quality.
• The numerical results show that the MM algorithm used in t-SNE is equivalent to DCA-Like applied to the t-SNE optimization problem.
• In t-SNE visualization on MNIST, all three algorithms (DCA, DCA-Like, ADCA-Like) preserve the data structure well, with ADCA-Like converging fastest and achieving the best objective value.