[Paper Review] Representation Learning and Recovery in the ReLU Model

Arya Mazumdar, Ankit Singh Rawat|arXiv (Cornell University)|Mar 12, 2018

Machine Learning and Algorithms15 references3 citations

TL;DR

This paper studies representation learning and robust signal recovery in a single-layer ReLU neural network model with random biases. It proves that the column space of the weight matrix can be recovered within O(d) Frobenius error, and a generalized LASSO algorithm achieves ℓ₂ recovery error of O(√((k+s)log d)/d) under Gaussian design, even with sparse outliers and dense noise.

ABSTRACT

Rectified linear units, or ReLUs, have become the preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a ReLU-network -- a neural network with ReLU activations. As a primarily theoretical study, we limit ourselves to a single-layer network. The first problem we study corresponds to dictionary-learning in the presence of nonlinearity (modeled by the ReLU functions). Given a set of observation vectors $\mathbf{y}^i \in \mathbb{R}^d, i =1, 2, \dots , n$, we aim to recover $d imes k$ matrix $A$ and the latent vectors $\{\mathbf{c}^i\} \subset \mathbb{R}^k$ under the model $\mathbf{y}^i = \mathrm{ReLU}(A\mathbf{c}^i +\mathbf{b})$, where $\mathbf{b}\in \mathbb{R}^d$ is a random bias. We show that it is possible to recover the column space of $A$ within an error of $O(d)$ (in Frobenius norm) under certain conditions on the probability distribution of $\mathbf{b}$. The second problem we consider is that of robust recovery of the signal in the presence of outliers, i.e., large but sparse noise. In this setting we are interested in recovering the latent vector $\mathbf{c}$ from its noisy nonlinear sketches of the form $\mathbf{v} = \mathrm{ReLU}(A\mathbf{c}) + \mathbf{e}+\mathbf{w}$, where $\mathbf{e} \in \mathbb{R}^d$ denotes the outliers with sparsity $s$ and $\mathbf{w} \in \mathbb{R}^d$ denote the dense but small noise. This line of work has recently been studied (Soltanolkotabi, 2017) without the presence of outliers. For this problem, we show that a generalized LASSO algorithm is able to recover the signal $\mathbf{c} \in \mathbb{R}^k$ within an $\ell_2$ error of $O(\sqrt{\frac{(k+s)\log d}{d}})$ when $A$ is a random Gaussian matrix.

Motivation & Objective

• To address the problem of dictionary learning in the presence of ReLU nonlinearity, where observations follow y^i = ReLU(A c^i + b) with random bias b.
• To study robust recovery of latent vectors c from nonlinear sketches corrupted by sparse outliers e and dense noise w, i.e., v = ReLU(A c) + e + w.
• To establish theoretical guarantees for both representation recovery and signal recovery under a generative ReLU model with a single-layer network.
• To extend the theoretical understanding of ReLU-based generative models, particularly in the context of associative memory and nonlinear sketching.
• To analyze the interplay between nonlinearity (ReLU), random bias, and noise in learning and recovery tasks, focusing on high-dimensional asymptotics.

Proposed method

• Uses a generative model where observations are nonlinear sketches of latent vectors via ReLU-activated linear transformations with random bias b.
• Applies a generalized LASSO formulation to recover the latent vector c from noisy observations v = ReLU(A c) + e + w, with sparsity s for outliers e.
• Employs a dual certificate approach and restricted strong convexity to bound the estimation error of the LASSO solution.
• Derives probabilistic bounds on the error using concentration inequalities, assuming A is an i.i.d. Gaussian matrix.
• Introduces a restricted set R to characterize the feasible solution space and uses Lemma 3 to lower-bound the quadratic form ‖A h + f‖²₂.
• Combines bounds on the dual norm and noise terms to derive a final ℓ₂ error bound on the recovery of c.

Experimental results

Research questions

• RQ1 Can the column space of the weight matrix A be recovered from nonlinear sketches y^i = ReLU(A c^i + b) when the bias b is random?
• RQ2 What is the fundamental limit of robust recovery of the latent vector c when the observation is corrupted by sparse outliers and dense noise?
• RQ3 How does the nonlinearity introduced by ReLU affect the recovery performance in high-dimensional settings?
• RQ4 Can a generalized LASSO approach achieve consistent recovery in the presence of both sparse and dense noise in ReLU-based sketches?
• RQ5 What conditions on the bias distribution and design matrix A ensure stable and accurate representation learning and signal recovery?

Key findings

• The column space of the weight matrix A can be recovered within an O(d) Frobenius norm error under mild conditions on the bias distribution.
• For robust recovery, a generalized LASSO algorithm achieves an ℓ₂ error bound of O(√((k + s) log d)/d) when A is a random Gaussian matrix.
• The error bound scales with √(k log k / d) for the nonlinearity-induced noise and √(s log d / d) for sparse outliers, reflecting the trade-off between latent dimension and sparsity.
• The recovery error is stable under high-dimensional asymptotics, with high probability, provided the number of measurements d is sufficiently large relative to k and s.
• The analysis establishes that the dual certificate method can be adapted to nonlinear models with ReLU activation, enabling theoretical guarantees for non-convex sketching.
• The bound holds with high probability when the noise ∥w∥∞ ≤ δ and A is i.i.d. Gaussian, ensuring robustness to dense but bounded noise.