[Paper Review] Scalable Semi-Supervised Learning over Networks using Nonsmooth Convex Optimization

Alexander Jung, Alfred O. Hero+2 more|arXiv (Cornell University)|Nov 2, 2016

Complex Network Analysis Techniques14 references3 citations

TL;DR

This paper proposes a scalable semi-supervised learning method for large network-structured datasets by modeling the hypothesis as a graph signal with low total variation, using nonsmooth convex optimization via Nesterov's optimal first-order method. The approach enables efficient, distributed message-passing implementation suitable for big data platforms, achieving high scalability while preserving label smoothness across clusters.

ABSTRACT

We propose a scalable method for semi-supervised (transductive) learning from massive network-structured datasets. Our approach to semi-supervised learning is based on representing the underlying hypothesis as a graph signal with small total variation. Requiring a small total variation of the graph signal representing the underlying hypothesis corresponds to the central smoothness assumption that forms the basis for semi-supervised learning, i.e., input points forming clusters have similar output values or labels. We formulate the learning problem as a nonsmooth convex optimization problem which we solve by appealing to Nesterovs optimal first-order method for nonsmooth optimization. We also provide a message passing formulation of the learning method which allows for a highly scalable implementation in big data frameworks.

Motivation & Objective

• To address the challenge of scalable semi-supervised learning on massive network-structured datasets with limited labeled data.
• To model the underlying hypothesis as a graph signal with small total variation to enforce smoothness across connected nodes.
• To develop an efficient, distributed optimization framework suitable for big data platforms like AKKA.
• To enable scalable learning by formulating the problem as a nonsmooth convex optimization task with a message-passing interface.
• To extend the framework to handle diverse loss functions beyond mean squared error for categorical or discrete labels.

Proposed method

• Formulates semi-supervised learning as a nonsmooth convex optimization problem with a total variation penalty on the graph signal.
• Applies Nesterov’s optimal first-order method to solve the nonsmooth problem efficiently, ensuring convergence guarantees.
• Derives a message-passing formulation where each node updates its label based on local information and neighboring signals.
• Introduces a dual variable update rule based on error bounds, enabling adaptive constraint handling in the optimization.
• Uses the graph Laplacian and total variation to encode smoothness, favoring cluster-wise label consistency over pairwise similarity.
• Employs a projected subgradient method with a constraint set defined by empirical error bounds to maintain feasibility.

Experimental results

Research questions

• RQ1 How can semi-supervised learning be made scalable for massive network-structured datasets with limited labeled data?
• RQ2 Can total variation regularization on graph signals better capture community structure than traditional Laplacian-based smoothness measures?
• RQ3 What is the optimal first-order method for solving nonsmooth convex optimization problems arising in graph-based semi-supervised learning?
• RQ4 How can the optimization be reformulated as a distributed message-passing algorithm for big data platforms?
• RQ5 What is the convergence behavior and computational complexity of the proposed method under large-scale settings?

Key findings

• The proposed method achieves scalable semi-supervised learning by leveraging nonsmooth convex optimization with total variation regularization on graph signals.
• The message-passing formulation enables distributed, local computation, making it suitable for implementation in big data frameworks like AKKA.
• The algorithm converges to the optimal solution using Nesterov’s optimal first-order method, ensuring fast convergence rates for nonsmooth problems.
• The dual variable update rule ensures constraint satisfaction by dynamically adjusting based on error bounds, maintaining feasibility throughout iterations.
• The method outperforms traditional label propagation by better aligning with community structure through total variation, which favors piecewise-constant signals over smooth ones.
• Theoretical analysis confirms that the solution satisfies KKT conditions and that the optimal dual variable is determined by the error in the current iterate.