[Paper Review] Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Mario Lezcano-Casado, David Martı́nez-Rubio|arXiv (Cornell University)|Jan 24, 2019

Model Reduction and Neural Networks38 references59 citations

TL;DR

The paper introduces an exponential-map based parametrization to optimize over orthogonal and unitary groups, enabling unconstrained first-order optimization for RNNs with robust, efficient training and competitive results. It demonstrates an exprnn architecture and provides implementation details and empirical comparisons against existing methods.

ABSTRACT

We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.

Motivation & Objective

• Motivate robust optimization under orthogonal/unitary constraints to mitigate exploding/vanishing gradients in RNNs.
• Propose a Lie-theoretic exponential parametrization to transform constrained problems into unconstrained Euclidean ones.
• Provide implementation tricks enabling practical, low-overhead use in neural networks.
• Demonstrate empirical advantages on standard long-sequence tasks compared to prior methods.

Proposed method

• Parametrize orthogonal/unitary groups G (SO(n), U(n)) via the matrix exponential exp(A) with A in the Lie algebra (skew-symmetric/skew-Hermitian).
• Show surjectivity of the exponential map on connected compact Lie groups and discuss metric changes induced by the parametrization.
• Use Padé approximants with scale-squaring to compute exp(A) and its gradient efficiently.
• Derive exact gradient formulas for f(exp(A)) to enable machine-precision gradients (Proposition 4.1).
• Define the exponential RNN (exprnn) with h_{t+1}=sigma(exp(A) h_t + T x_{t+1}), A skew-symmetric, T a linear map.
• Discuss initialization and practical truncations (retractions) for optimization.

Experimental results

Research questions

• RQ1 Can a matrix exponential parametrization provide a robust, cheap alternative to hard orthogonality constraints in neural networks?
• RQ2 Does optimizing over skew-symmetric matrices via the exponential map improve convergence, stability, and generalization for RNNs with orthogonal constraints?
• RQ3 How does the exponential parametrization compare to existing methods (Cayley transform, Riemannian gradient methods) in practice?
• RQ4 What are practical tricks to implement and initialize such parametrizations efficiently in neural networks?

Key findings

• Exponential parametrization yields faster, more stable convergence for orthogonal RNNs (exprnn) across multiple tasks.
• Parametrization enables use of general optimizers without enforcing hard orthogonality, with negligible runtime overhead in practice.
• Empirical results show exprnn surpasses or matches competing methods on copying memory, pixel-MNIST, and timit speech tasks across several model sizes.
• Padé-based exponential approximations with scale-squaring provide machine-precision exp and exact gradient computations, improving numerical stability.
• Initialization strategies (block-diagonal skew-symmetric blocks) help leverage eigenvalue structure for better training dynamics.
• The approach achieves competitive or superior performance while avoiding degeneracies seen in some Cayley-transform methods.