[论文解读] Riemannian Adaptive Optimization Methods

Gary Bécigneul, Octavian-Eugen Ganea|arXiv (Cornell University)|Oct 1, 2018

Stochastic Gradient Optimization Techniques参考文献 29被引用 93

一句话总结

本文将自适应优化方法（Adagrad、Adam、Amsgrad）推广到黎曼乘积流形上，给出曲率测地凸目标的收敛性证明，并在超球面嵌入的经验性结果中展示收益。

ABSTRACT

Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools - namely Adam , Adagrad and the more recent Amsgrad - remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincare ball.

研究动机与目标

• Explain the challenges of creating intrinsic adaptive optimizers on general Riemannian manifolds.
• Propose Riemannian versions of Adagrad, Adam, and Amsgrad for Cartesian products of manifolds.
• Provide convergence analysis for geodesically convex objectives on product manifolds.
• Empirically validate the methods on hyperbolic (Poincaré ball) taxonomy embedding tasks.

提出的方法

• Formulate adaptive updates across manifold components in a product manifold setting (x = (x1,...,xn)).
• Define per-component gradient norms using Riemannian metrics to scale updates (||g_t^i||_{x_t^i}).
• Derive Ramsgrad and RadamNc algorithms on product manifolds with intrinsic exponential maps and parallel transport.
• Prove regret bounds and convergence guarantees in the geodesically convex setting, incorporating curvature via a zeta term.
• Compare with Euclidean Adagrad/Adam/Amsgrad and discuss special cases where Euclidean results are recovered.
• Experiment with hyperbolic WordNet embeddings in the Poincaré ball using retraction and exponential-map updates.

实验结果

研究问题

• RQ1 How can adaptive optimization methods be extended to Riemannian manifolds in an intrinsic way?
• RQ2 Can adaptivity be meaningfully implemented across coordinates on a product of manifolds?
• RQ3 Do Riemannian versions of Adagrad/Adam/Amsgrad provide convergence guarantees and practical benefits?
• RQ4 How do curvature and manifold geometry affect convergence and performance of these adaptive methods?
• RQ5 Are the proposed Riemannian adaptive methods advantageous for non-Euclidean embedding tasks such as hyperbolic taxonomy embedding?

主要发现

• Riemannian Adagrad, Ramsgrad, and RadamNc are feasible on Cartesian product manifolds with per-component adaptive updates.
• Convergence guarantees (regret bounds) are established for Ramsgrad and RadamNc on geodesically convex objectives, with curvature-dependent terms.
• The curvature of the manifolds appears in the bounds through a zeta term, interpolating between Euclidean and curved cases.
• Empirical results on hyperbolic WordNet embeddings show faster convergence and lower train loss for Riemannian adaptive methods compared to non-adaptive baselines.
• In retraction-based experiments, Radam achieves the lowest training loss, while Ramsgrad can generalize better on link prediction tasks.