[Paper Review] Sparse Bilinear Logistic Regression

Shi, Jianing V., Yangyang Xu+1 more|arXiv (Cornell University)|Apr 15, 2014

Sparse and Compressive Sensing Techniques41 references20 citations

TL;DR

This paper proposes sparse bilinear logistic regression, a method that models matrix-shaped explanatory variables using low-rank factorizations of weight matrices to enable feature selection and improved generalization. By combining bilinear modeling with sparsity-inducing regularization via block coordinate descent with proximal updates, the approach achieves global convergence and outperforms existing methods in computer vision, EEG-based brain-computer interfaces, and style/content factorization tasks.

ABSTRACT

In this paper, we introduce the concept of sparse bilinear logistic regression for decision problems involving explanatory variables that are two-dimensional matrices. Such problems are common in computer vision, brain-computer interfaces, style/content factorization, and parallel factor analysis. The underlying optimization problem is bi-convex; we study its solution and develop an efficient algorithm based on block coordinate descent. We provide a theoretical guarantee for global convergence and estimate the asymptotical convergence rate using the Kurdyka-Łojasiewicz inequality. A range of experiments with simulated and real data demonstrate that sparse bilinear logistic regression outperforms current techniques in several important applications.

Motivation & Objective

• To address classification problems where explanatory variables are two-dimensional matrices, such as images, video frames, or EEG signals.
• To extend sparse logistic regression to bilinear models that preserve matrix structure and enable dimensionality reduction.
• To develop an efficient optimization algorithm that ensures global convergence despite the bi-convex nature of the problem.
• To promote sparsity in the spatial and temporal factors of the bilinear model for improved interpretability and feature selection.
• To empirically validate the method's superiority over standard logistic regression and bilinear models in real-world applications.

Proposed method

• Formulates the sparse bilinear logistic regression problem as minimizing a log-likelihood loss with nuclear norm and sparsity-inducing regularizers on the factor matrices.
• Uses block coordinate descent with proximal updates to solve subproblems, accelerating convergence compared to standard alternating minimization.
• Applies the Kurdyka-Łojasiewicz inequality to establish global convergence and estimate asymptotic convergence rates.
• Represents the decision boundary via a low-rank weight matrix W = UV^T, where U and V are learned spatial and temporal factors.
• Implements a variational formulation that decouples the optimization over factor matrices while maintaining the bilinear structure.
• Generalizes the model to multinomial classification by extending the bilinear form to multiple class-specific factor matrices.

Experimental results

Research questions

• RQ1 Can sparsity be effectively incorporated into bilinear logistic regression to improve generalization and feature selection for matrix-valued data?
• RQ2 Does the proposed block coordinate descent with proximal updates ensure global convergence for the bi-convex sparse bilinear model?
• RQ3 How does the asymptotic convergence rate of the algorithm behave, and can it be theoretically bounded using the Kurdyka-Łojasiewicz inequality?
• RQ4 In what applications does sparse bilinear logistic regression outperform standard logistic regression and unregularized bilinear models?
• RQ5 Can the bilinear factorization with sparsity promote better interpretability and performance in tasks like style/content separation and EEG-based classification?

Key findings

• The proposed algorithm achieves global convergence and provides a theoretical convergence rate estimate using the Kurdyka-Łojasiewicz inequality.
• Sparse bilinear logistic regression outperforms standard logistic regression and unregularized bilinear models in simulated and real-world classification tasks.
• The method improves generalization performance in EEG-based brain-computer interfaces, where matrix-structured data are common.
• Empirical results show that sparsity in the factor matrices leads to better feature selection and improved model interpretability.
• The multinomial extension of the model successfully generalizes to multi-class classification with matrix inputs.
• Experiments demonstrate that the approach surpasses PCA-based dimensionality reduction and standard SVMs in bilinear feature spaces.