[Paper Review] Hyperspherical Prototype Networks

Pascal Mettes, Elise van der Pol+1 more|arXiv (Cornell University)|Jan 29, 2019

Modular Robots and Swarm Intelligence46 references50 citations

TL;DR

The paper defines fixed hyperspherical class prototypes and uses cosine similarity to map inputs for classification and regression, enabling multi-task learning in a scalable output space without updating prototypes.

ABSTRACT

This paper introduces hyperspherical prototype networks, which unify classification and regression with prototypes on hyperspherical output spaces. For classification, a common approach is to define prototypes as the mean output vector over training examples per class. Here, we propose to use hyperspheres as output spaces, with class prototypes defined a priori with large margin separation. We position prototypes through data-independent optimization, with an extension to incorporate priors from class semantics. By doing so, we do not require any prototype updating, we can handle any training size, and the output dimensionality is no longer constrained to the number of classes. Furthermore, we generalize to regression, by optimizing outputs as an interpolation between two prototypes on the hypersphere. Since both tasks are now defined by the same loss function, they can be jointly trained for multi-task problems. Experimentally, we show the benefit of hyperspherical prototype networks for classification, regression, and their combination over other prototype methods, softmax cross-entropy, and mean squared error approaches.

Motivation & Objective

• Introduce a class of networks with hyperspherical output spaces and fixed prototypes.
• Enable large-margin separation and semantic priors in prototype placement.
• Unify classification and regression under a common cosine-similarity loss.
• Allow flexible output dimensionality beyond the number of classes.
• Demonstrate benefits for classification, regression, and multi-task settings.

Proposed method

• Place K class prototypes on the D-dimensional unit hypersphere prior to training to enforce large-margin separation.
• Train by minimizing (1 - cos theta)^2 between outputs and the ground-truth class prototype using the network f_phi(x).
• Position prototypes through data-independent optimization, with optional semantic priors via a ranking-based loss over class triplets.
• Extend to regression by maintaining two opposite prototypes as upper and lower bounds and interpolating via cosine similarity.
• Permit joint classification and regression by using the same squared cosine loss without task weighting.
• Provide a mechanism to optimize prototype placement using a differentiable objective that enforces large separation on the hypersphere.

Experimental results

Research questions

• RQ1 Can fixed hyperspherical prototypes provide competitive accuracy across standard datasets with variable output dimensionality?
• RQ2 Does incorporating semantic priors into prototype placement improve classification when output space is small or unevenly distributed?
• RQ3 Can regression be effectively performed in a higher-dimensional hyperspherical output space using cosine-based interpolation between bounds?
• RQ4 Is it possible to train classification and regression jointly in the same output space without tuning task weights?
• RQ5 How do hyperspherical prototypes compare to传统 prototype-based methods and softmax in terms of convergence and robustness?

Key findings

• Hyperspherical prototypes achieve high accuracy across CIFAR-100 and ImageNet-200 for multiple output dimensions, outperforming one-hot and Word2vec prototype baselines.
• Prototype separation on the hypersphere is consistently higher than baselines, indicating better class separation.
• Incorporating privileged semantic information improves classification, especially at smaller output dimensions.
• The approach yields robust regression results on OmniArt with higher resilience to learning rate choices than baseline MSE methods.
• Joint regression and classification on rotated MNIST demonstrates the same-space modeling capability for both tasks.
• Compared to softmax cross-entropy, hyperspherical prototypes perform better when training data per class is scarce or uneven.