[Paper Review] Orthogonal Over-Parameterized Training

Weiyang Liu, Rongmei Lin+6 more|arXiv (Cornell University)|Apr 9, 2020

Advanced Neural Network Applications81 references4 citations

TL;DR

This paper proposes Orthogonal Over-Parameterized Training (OPT), a novel framework that improves neural network generalization by learning an orthogonal transformation to maintain minimal hyperspherical energy—quantifying neuron diversity on a hypersphere. By fixing randomly initialized neuron weights and optimizing a shared orthogonal matrix, OPT provably minimizes energy and generalizes better than standard training.

ABSTRACT

The inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. By maintaining the minimum hyperspherical energy during training, OPT can greatly improve the empirical generalization. Specifically, OPT fixes the randomly initialized weights of the neurons and learns an orthogonal transformation that applies to these neurons. We consider multiple ways to learn such an orthogonal transformation, including unrolling orthogonalization algorithms, applying orthogonal parameterization, and designing orthogonality-preserving gradient descent. For better scalability, we propose the stochastic OPT which performs orthogonal transformation stochastically for partial dimensions of neurons. Interestingly, OPT reveals that learning a proper coordinate system for neurons is crucial to generalization. We provide some insights on why OPT yields better generalization. Extensive experiments validate the superiority of OPT over the standard training.

Motivation & Objective

• To improve generalization in over-parameterized neural networks by introducing a new inductive bias through orthogonal parameterization.
• To address the gap in existing training methods that fail to explicitly minimize hyperspherical energy despite its known link to generalization.
• To decouple neuron weight learning from coordinate system learning by fixing initial weights and optimizing an orthogonal transformation.
• To provide a scalable, provably effective method for minimizing hyperspherical energy during training via orthogonal matrix optimization.
• To reveal that learning a proper coordinate system for neurons is crucial for generalization, not just weight updates.

Proposed method

• Decomposes each neuron's weight as w = Rv, where v is randomly initialized and fixed, and R is a learnable layer-shared orthogonal matrix.
• Uses unrolled orthogonalization algorithms (e.g., Gram-Schmidt, Householder, Löwdin) to learn R with implicit regularization.
• Applies orthogonal parameterization (e.g., Cayley parametrization) for efficient and stable optimization of R.
• Employs orthogonality-preserving gradient descent to maintain R’s orthogonality during backpropagation.
• Introduces stochastic OPT that applies orthogonal transformations only to partial neuron dimensions for improved scalability.
• Preprocesses random weights by minimizing their hyperspherical energy before training to enhance initial neuron separation.

Experimental results

Research questions

• RQ1 Can learning an orthogonal transformation for neuron weights provably minimize hyperspherical energy and improve generalization?
• RQ2 How does decoupling weight magnitude from coordinate system learning affect inductive bias and generalization?
• RQ3 What are the relative benefits of different orthogonalization methods (e.g., unrolled, parameterized, gradient-preserving) in the OPT framework?
• RQ4 Does minimizing hyperspherical energy of randomly initialized weights before training improve downstream performance?
• RQ5 How does stochastic OPT maintain performance while improving scalability for large models?

Key findings

• OPT provably minimizes hyperspherical energy by learning an orthogonal transformation R, ensuring minimal energy in expectation when v is randomly initialized.
• Experiments show OPT generalizes significantly better than standard training, even when training loss is similar.
• The framework improves generalization by learning a coordinate system (R) that enhances neuron diversity and separation on the hypersphere.
• Preprocessing random weights to minimize their hyperspherical energy leads to better initial configurations and faster convergence.
• Stochastic OPT maintains strong performance while reducing computational cost, enabling scalability to large models.
• Different orthogonalization methods (e.g., symmetric vs. Householder) induce distinct implicit regularizations, affecting final model performance.