[Paper Review] Nonconvex One-bit Single-label Multi-label Learning

Shuang Qiu, Tingjin Luo+2 more|arXiv (Cornell University)|Mar 17, 2017

Sparse and Compressive Sensing Techniques20 references3 citations

TL;DR

This paper proposes a non-convex alternating power iteration algorithm for one-bit single-label multi-label learning, formulating it as a rank-one matrix sensing problem with heterogeneous sensing vectors and signum link function. The method achieves linear convergence and near-optimal sampling complexity of $ O(k^{1.5}d_1d_2/\epsilon) $ for $ O(\epsilon) $ recovery error, significantly improving state-of-the-art sampling efficiency in extreme low-label regimes.

ABSTRACT

We study an extreme scenario in multi-label learning where each training instance is endowed with a single one-bit label out of multiple labels. We formulate this problem as a non-trivial special case of one-bit rank-one matrix sensing and develop an efficient non-convex algorithm based on alternating power iteration. The proposed algorithm is able to recover the underlying low-rank matrix model with linear convergence. For a rank-$k$ model with $d_1$ features and $d_2$ classes, the proposed algorithm achieves $O(ε)$ recovery error after retrieving $O(k^{1.5}d_1 d_2/ε)$ one-bit labels within $O(kd)$ memory. Our bound is nearly optimal in the order of $O(1/ε)$. This significantly improves the state-of-the-art sampling complexity of one-bit multi-label learning. We perform experiments to verify our theory and evaluate the performance of the proposed algorithm.

Motivation & Objective

• Address the challenge of learning low-rank multi-label models when each instance is labeled with only one binary label out of many possible classes.
• Develop a provably efficient algorithm for one-bit multi-label learning under extreme label scarcity, where only one label per instance is observed.
• Overcome the limitations of existing convex or symmetric sensing-based methods by handling heterogeneous sensing vectors and non-smooth signum link functions.
• Achieve near-optimal sampling complexity and linear convergence for recovering the underlying low-rank weight matrix in high-dimensional settings.
• Verify theoretical guarantees through experiments on synthetic data under noise and full-observation settings.

Proposed method

• Formulate the single-label multi-label learning problem as a non-convex one-bit rank-one matrix sensing problem with asymmetric sensing: left vector is i.i.d. Gaussian, right vector is one-hot sampled.
• Propose an alternating power iteration algorithm to estimate the low-rank factor matrices, avoiding convex relaxation and enabling efficient computation.
• Use noisy power iteration to construct estimation sequences, enabling convergence analysis despite non-convexity and non-smooth link functions.
• Incorporate the signum (Hamming) loss directly into the model, avoiding surrogate losses used in prior work.
• Ensure memory efficiency with $ O(kd) $ space complexity, where $ d = d_1 + d_2 $, and scale to large $ d_1, d_2 $.
• Apply theoretical analysis based on estimation sequences and sub-Gaussian concentration to establish linear convergence and recovery error bounds.

Experimental results

Research questions

• RQ1 Can a non-convex algorithm achieve linear convergence in one-bit single-label multi-label learning with heterogeneous sensing vectors?
• RQ2 What is the optimal sampling complexity required to recover a low-rank weight matrix under one-bit label observations?
• RQ3 How does the proposed algorithm perform under noisy or adversarial label perturbations compared to existing methods?
• RQ4 Can the algorithm be extended to full-observation multi-label learning settings with comparable or better performance?
• RQ5 Does the use of the signum loss function enable better generalization than convex surrogate losses in this extreme labeling regime?

Key findings

• The proposed algorithm achieves $ O(\epsilon) $ recovery error after retrieving $ O(k^{1.5}d_1d_2/\epsilon) $ one-bit labels, which is nearly optimal in the $ O(1/\epsilon) $ order.
• The algorithm converges linearly and operates within $ O(kd) $ memory, making it scalable to high-dimensional feature and class spaces.
• Experiments show the algorithm outperforms LEML in both prediction error and AUC across all noise settings, including adversarial noise and random perturbations.
• On synthetic full-observation datasets, the method achieves average AUC of 98.15 on the largest dataset, surpassing all baselines including LEML, SVM, LRT, and SMT.
• The algorithm maintains robustness under varying noise levels ($ \xi = 0.1, 0.2, 0.3 $) and label sparsity ($ p = 1\%, 2.5\%, 5\% $), consistently outperforming LEML.
• Theoretical analysis confirms global convergence despite non-convexity and non-smoothness, overcoming limitations of prior convex or symmetric sensing frameworks.