[Paper Review] Multi-view Laplacian Support Vector Machines

Shiliang Sun|arXiv (Cornell University)|Jul 26, 2013

Face and Expression Recognition25 references4 citations

TL;DR

This paper proposes Multi-view Laplacian Support Vector Machines (MvLapSVM), a novel semi-supervised learning method that integrates manifold regularization and multi-view regularization into SVMs for multi-view data. By optimizing a convex dual problem in a reproducing kernel Hilbert space and leveraging empirical Rademacher complexity, the method achieves improved generalization and outperforms existing approaches on synthetic and real-world datasets, including NBA-NASCAR and web page categorization tasks.

ABSTRACT

We propose a new approach, multi-view Laplacian support vector machines (SVMs), for semi-supervised learning under the multi-view scenario. It integrates manifold regularization and multi-view regularization into the usual formulation of SVMs and is a natural extension of SVMs from supervised learning to multi-view semi-supervised learning. The function optimization problem in a reproducing kernel Hilbert space is converted to an optimization in a finite-dimensional Euclidean space. After providing a theoretical bound for the generalization performance of the proposed method, we further give a formulation of the empirical Rademacher complexity which affects the bound significantly. From this bound and the empirical Rademacher complexity, we can gain insights into the roles played by different regularization terms to the generalization performance. Experimental results on synthetic and real-world data sets are presented, which validate the effectiveness of the proposed multi-view Laplacian SVMs approach.

Motivation & Objective

• To address the challenge of semi-supervised learning in multi-view data settings where labeled data is scarce but unlabeled data is abundant.
• To extend traditional SVMs by incorporating manifold regularization to exploit the intrinsic data structure and multi-view regularization to ensure consistency across views.
• To provide a theoretical foundation for generalization performance through empirical Rademacher complexity analysis.
• To empirically validate the effectiveness of the proposed method on both synthetic and real-world multi-view datasets.

Proposed method

• The method formulates a primal optimization problem in a reproducing kernel Hilbert space that combines three regularization terms: function norm, manifold (graph Laplacian), and multi-view regularization.
• Manifold regularization is implemented via a graph Laplacian matrix constructed from pairwise similarities between data points, promoting smoothness of the decision function over the data manifold.
• Multi-view regularization enforces consistency between predictions derived from individual views, using a combined loss function that depends on predictions from each view separately.
• The primal problem is transformed into a dual optimization problem in a finite-dimensional Euclidean space, enabling efficient solution via convex optimization techniques.
• Theoretical analysis includes deriving a generalization error bound and analyzing the role of each regularization term through empirical Rademacher complexity.
• The method uses a normalized graph Laplacian (Laplacian matrix) to improve stability and performance in practice.

Experimental results

Research questions

• RQ1 How can manifold regularization be effectively combined with multi-view regularization in a semi-supervised SVM framework to improve generalization?
• RQ2 What is the theoretical impact of each regularization term—function norm, manifold, and multi-view—on the generalization error of the model?
• RQ3 How does the empirical Rademacher complexity reflect the influence of different regularization components on model capacity and performance?
• RQ4 Can the proposed method outperform existing state-of-the-art methods such as LapSVM, Co-Laplacian SVM, and Co-SVM in multi-view semi-supervised learning?
• RQ5 To what extent does the integration of multiple views and manifold structure enhance performance on real-world datasets with limited labeled examples?

Key findings

• On the NBA-NASCAR dataset, MvLapSVM achieved a test accuracy of 99.38% (±0.64) and an unlabeled set accuracy of 99.54% (±0.56), outperforming all baselines.
• On the web page categorization dataset, MvLapSVM achieved 94.25% (±1.62) on the test set and 93.53% (±2.04) on the unlabeled training set, surpassing all compared methods.
• Theoretical analysis revealed that multi-view regularization plays a significant role in reducing the empirical Rademacher complexity, thereby improving generalization performance.
• The method demonstrated strong generalization due to the balanced integration of manifold and multi-view regularization, as validated by the Rademacher complexity analysis.
• Empirical results confirm that MvLapSVM consistently ranks first or near the top across multiple datasets and evaluation metrics, indicating robustness and effectiveness.