[Paper Review] Free Hyperbolic Neural Networks with Limited Radii.

Yunhui Guo, Xudong Wang+2 more|arXiv (Cornell University)|Jul 23, 2021

Advanced Neural Network Applications18 references4 citations

TL;DR

This paper proposes Feature Clipping, a regularization technique that constrains hyperbolic embeddings within a limited radius to mitigate the vanishing gradient problem in Hyperbolic Neural Networks (HNNs). By stabilizing training, the method enables HNNs to achieve performance comparable to Euclidean Neural Networks on standard image benchmarks like CIFAR10, CIFAR100, and ImageNet, while improving adversarial robustness and out-of-distribution detection.

ABSTRACT

Non-Euclidean geometry with constant negative curvature, i.e., hyperbolic space, has attracted sustained attention in the community of machine learning. Hyperbolic space, owing to its ability to embed hierarchical structures continuously with low distortion, has been applied for learning data with tree-like structures. Hyperbolic Neural Networks (HNNs) that operate directly in hyperbolic space have also been proposed recently to further exploit the potential of hyperbolic representations. While HNNs have achieved better performance than Euclidean neural networks (ENNs) on datasets with implicit hierarchical structure, they still perform poorly on standard classification benchmarks such as CIFAR and ImageNet. The traditional wisdom is that it is critical for the data to respect the hyperbolic geometry when applying HNNs. In this paper, we first conduct an empirical study showing that the inferior performance of HNNs on standard recognition datasets can be attributed to the notorious vanishing gradient problem. We further discovered that this problem stems from the hybrid architecture of HNNs. Our analysis leads to a simple yet effective solution called Feature Clipping, which regularizes the hyperbolic embedding whenever its norm exceeding a given threshold. Our thorough experiments show that the proposed method can successfully avoid the vanishing gradient problem when training HNNs with backpropagation. The improved HNNs are able to achieve comparable performance with ENNs on standard image recognition datasets including MNIST, CIFAR10, CIFAR100 and ImageNet, while demonstrating more adversarial robustness and stronger out-of-distribution detection capability.

Motivation & Objective

• To investigate why Hyperbolic Neural Networks (HNNs) underperform on standard image recognition benchmarks despite their theoretical advantages for hierarchical data.
• To identify the root cause of poor HNN performance on non-hierarchical datasets such as CIFAR and ImageNet.
• To develop a simple yet effective regularization method that stabilizes training in HNNs by preventing gradient vanishing.
• To enable HNNs to achieve competitive performance on standard vision benchmarks while retaining benefits like adversarial robustness and out-of-distribution detection.

Proposed method

• Introduce Feature Clipping, a regularization technique that clips the hyperbolic norm of embeddings whenever it exceeds a predefined threshold.
• Apply the clipping operation during backpropagation to prevent gradients from vanishing in deep HNNs.
• Maintain the intrinsic geometry of hyperbolic space while constraining the radial extent of representations to a limited radius.
• Use the Poincaré ball model for hyperbolic space, where the clipping is applied to the hyperbolic norm of hidden representations.
• Integrate Feature Clipping into standard HNN architectures without modifying the network depth or width.
• Train HNNs end-to-end using standard backpropagation with the clipped embeddings to stabilize optimization.

Experimental results

Research questions

• RQ1 Why do Hyperbolic Neural Networks underperform on standard image recognition datasets like CIFAR10 and ImageNet?
• RQ2 What causes the vanishing gradient problem in HNNs during training on non-hierarchical data?
• RQ3 Can the vanishing gradient issue in HNNs be mitigated through a simple architectural regularization technique?
• RQ4 Can HNNs with limited radial extent achieve performance comparable to Euclidean Neural Networks on standard vision benchmarks?
• RQ5 Do HNNs with Feature Clipping maintain or improve upon advantages like adversarial robustness and out-of-distribution detection?

Key findings

• The vanishing gradient problem in HNNs is primarily caused by the hybrid architecture, where hyperbolic and Euclidean components interact unstably during backpropagation.
• Feature Clipping effectively prevents gradient vanishing by constraining the radial extent of hyperbolic embeddings to a limited radius.
• HNNs with Feature Clipping achieve performance on par with Euclidean Neural Networks on MNIST, CIFAR10, CIFAR100, and ImageNet.
• The improved HNNs demonstrate stronger adversarial robustness compared to standard ENNs on benchmark datasets.
• The HNNs with Feature Clipping show superior out-of-distribution detection capability, as measured by AUROC scores on OOD benchmarks.
• The method is simple, effective, and generalizable, requiring no architectural changes beyond the clipping operation.