[Paper Review] Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing

Hiroyuki Sakai, Hideaki Iiduka|arXiv (Cornell University)|Apr 2, 2020

Stochastic Gradient Optimization Techniques20 references4 citations

TL;DR

This paper proposes a Riemannian adaptive optimization algorithm that extends AMSGrad to Riemannian manifolds, enabling direct solution of Riemannian stochastic optimization problems with constant learning rates. The method achieves faster and more stable convergence than existing algorithms in Poincaré embeddings and principal component analysis, particularly outperforming RSGD, RAdaGrad, and RAdam on WordNet hierarchy tasks.

ABSTRACT

This paper proposes a Riemannian adaptive optimization algorithm to optimize the parameters of deep neural networks. The algorithm is an extension of both AMSGrad in Euclidean space and RAMSGrad on a Riemannian manifold. The algorithm helps to resolve two issues affecting RAMSGrad. The first is that it can solve the Riemannian stochastic optimization problem directly, in contrast to RAMSGrad which only achieves a low regret. The other is that it can use constant learning rates, which makes it implementable in practice. Additionally, we apply the proposed algorithm to Poincar{é} embeddings, which embed the transitive closure of the WordNet nouns into the Poincar{é} ball model of hyperbolic space. Numerical experiments show that regardless of the initial value of the learning rate, our algorithm stably converges to the optimal solution and converges faster than RSGD, the most basic Riemannian stochastic optimization algorithm.

Motivation & Objective

• To address the limitations of RAMSGrad, which only achieves regret minimization and requires diminishing learning rates.
• To develop a Riemannian adaptive optimization algorithm that directly solves the Riemannian optimization problem on manifolds.
• To enable practical deployment by supporting constant learning rates, unlike prior methods that become ineffective due to vanishing rates.
• To apply the algorithm to natural language processing tasks, particularly Poincaré embeddings for hierarchical data.
• To demonstrate superior convergence speed and stability compared to existing Riemannian optimization methods.

Proposed method

• Proposes a modified RAMSGrad algorithm (Algorithm 1) that extends Euclidean AMSGrad to Riemannian manifolds via Riemannian geometry operations, including exponential maps and parallel transport.
• Uses adaptive learning rates based on second-order momentum estimates, with reweighting via inverse scaling of the square root of the adaptive variance estimate.
• Applies the algorithm to the Poincaré ball model of hyperbolic space, where the manifold structure naturally models hierarchical data like WordNet.
• Employs a convergence analysis under standard assumptions, proving convergence for both constant and diminishing learning rates.
• Introduces a novel regret bound analysis that establishes direct convergence to the optimal solution, not just regret minimization.
• Uses a Riemannian version of the Adam-type update rule, incorporating retraction and vector transport to maintain iterates on the manifold.

Experimental results

Research questions

• RQ1 Can a Riemannian adaptive optimization algorithm be designed to directly solve the Riemannian optimization problem rather than only minimizing regret?
• RQ2 Can such an algorithm be made practical by using constant learning rates instead of diminishing ones?
• RQ3 Does the proposed algorithm converge faster and more stably than existing Riemannian adaptive methods like RSGD, RAdaGrad, and RAdam in Poincaré embedding tasks?
• RQ4 How does the choice between constant and diminishing learning rates affect performance in different learning problems like Poincaré embeddings and principal component analysis?
• RQ5 Can the algorithm achieve robust performance across different initial learning rate values in real-world NLP applications?

Key findings

• The proposed algorithm converges to the optimal solution faster than RSGD, RAdaGrad, and RAdam in Poincaré embedding tasks on the WordNet mammals subtree.
• The algorithm achieves stable convergence regardless of the initial learning rate value, demonstrating robustness to hyperparameter initialization.
• With constant learning rates, the algorithm outperforms existing methods in convergence speed and stability, particularly in hyperbolic space embeddings.
• The convergence analysis proves direct convergence to the optimal solution, not just regret minimization, under standard assumptions.
• The algorithm shows improved performance in principal component analysis, with convergence behavior depending on the dataset when using constant versus diminishing learning rates.
• The theoretical analysis establishes convergence for both constant and diminishing learning rates, with Corollary 3.1 confirming practical applicability via constant learning rates.