[Paper Review] Robust low-rank training via approximate orthonormal constraints

Dayana Savostianova, Emanuele Zangrando+2 more|arXiv (Cornell University)|Jun 2, 2023

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a robust low-rank training method that maintains model accuracy and efficiency while improving adversarial robustness by enforcing approximate orthonormal constraints on low-rank weight matrices. By projecting gradient flows onto the low-rank manifold and bounding singular values within a narrow band, the method ensures a bounded condition number, significantly enhancing robustness without increasing computational cost beyond standard low-rank training.

ABSTRACT

With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Motivation & Objective

• To address the trade-off between model efficiency and adversarial robustness in low-rank neural networks.
• To identify why low-rank factorization degrades robustness, linking it to exploding singular values and high condition numbers.
• To develop a training algorithm that maintains low-rank structure while enforcing bounded singular values to improve network conditioning.
• To prove theoretically that the method approximates an ideal low-rank model with bounded singular values, provided such a model exists.
• To demonstrate empirically that the method retains or improves both accuracy and robust accuracy across multiple architectures and datasets.

Proposed method

• Models robustness via the condition number of the neural network, linking poor robustness to high condition numbers caused by unbounded singular values in low-rank matrices.
• Derives three separate projected gradient flows for the low-rank factors $U$, $S$, and $V$ using geometric integration on manifolds to enforce constraints.
• Imposes approximate orthonormality by constraining the singular values of the low-rank matrices within a narrow band $[s - \varepsilon, s + \varepsilon]$ around a target value $s$.
• Uses continuous-time gradient flow formulations with orthogonal projections $P(Y)$ onto the tangent space of the low-rank manifold $\mathcal{M}_r$ to maintain the manifold structure.
• Applies a differential inequality framework to bound the error between the computed solution $Y(t)$ and the ideal low-rank path $\widetilde{W}(t)$, showing $\|Y(t) - W(t)\| \leq 3t\eta$ under bounded perturbations.
• Employs a first-order approximation theorem to prove that if a high-performing low-rank network with bounded singular values exists, the method computes it up to a small error.

Experimental results

Research questions

• RQ1 Why do low-rank neural networks exhibit reduced adversarial robustness compared to full-rank models?
• RQ2 Can low-rank training be made robust without sacrificing model accuracy or increasing training cost?
• RQ3 To what extent can approximate orthonormal constraints on low-rank factors improve the condition number and thus robustness?
• RQ4 Under what conditions does the proposed method approximate an optimal low-rank model with bounded singular values?
• RQ5 How does the method compare to existing baselines in terms of accuracy, robust accuracy, and computational efficiency across diverse architectures and datasets?

Key findings

• The proposed method achieves robust low-rank training with bounded singular values, significantly improving adversarial robustness without compromising accuracy.
• Empirical evaluations show that the robust low-rank networks match or exceed the robust accuracy of full-rank models on multiple datasets and architectures.
• The method reduces both training and inference costs by leveraging low-rank factorization while maintaining a bounded condition number through constrained singular values.
• Theoretical analysis confirms that if a high-performing low-rank network with bounded singular values exists, the algorithm computes it up to a first-order approximation error.
• The error between the computed model and the ideal low-rank path is bounded by $3t\eta$, ensuring stability and convergence under the projected gradient flow.
• The method is applicable to feed-forward networks and can be extended to convolutional layers by reshaping filters into matrices, as in prior low-rank methods.