[Paper Review] Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters

Aston Zhang, Yi Tay+5 more|arXiv (Cornell University)|Feb 17, 2021

Computational Physics and Python ApplicationsComputer Science17 references36 citations

TL;DR

The paper introduces parameterized hypercomplex multiplication (PHM) layers that learn hypercomplex multiplication rules from data, enabling arbitrary n-dimensional hypercomplex spaces with about 1/n the parameters of standard FC layers, and demonstrates them in LSTMs and Transformers.

ABSTRACT

Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, "fully-connected layers with Quaternions" (4D hypercomplex numbers), which replace real-valued matrix multiplications in fully-connected layers with Hamilton products of Quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists at very few predefined dimensions (4D, 8D, and 16D). This restricts the flexibility of models that leverage hypercomplex multiplications. To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary nD hypercomplex space, providing more architectural flexibility using arbitrarily $1/n$ learnable parameters compared with the fully-connected layer counterpart. Experiments of applications to the LSTM and Transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural flexibility and effectiveness of the proposed approach.

Motivation & Objective

• Motivate parameterizing hypercomplex multiplications to overcome fixed dimensionality limits (4D/8D/16D) in existing hypercomplex layers.
• Propose the PHM layer that expresses multiplication rules as a learned sum of Kronecker products, enabling arbitrary n-D hypercomplex spaces.
• Show that PHM layers reduce parameters while maintaining or improving performance in LSTM and Transformer architectures.
• Demonstrate applicability through natural language inference, machine translation, text style transfer, and subject-verb agreement tasks.

Proposed method

• Define the PHM layer that computes y = Hx + b with H constructed as a sum of Kronecker products H = sum_{i=1}^n A_i ⊗ S_i.
• Ensure k and d are divisible by a user-defined n, with A_i ∈ R^{n×n} and S_i ∈ R^{k/n × d/n}.
• Show that the parameter count of the PHM layer is approximately (kd)/n under mild assumptions, yielding ~1/n of a standard FC layer.
• Demonstrate that PHM subsumes real-valued matrix multiplication (n=1) and Hamilton products in Quaternion space (n=4), and can generalize to octonions/sedenions (n=8,16).
• Explain how PHM can express predefined hypercomplex rules (e.g., Hamilton product) by setting A_i and S_i accordingly and learn new multiplication interactions from data.

Experimental results

Research questions

• RQ1 Can a learnable hypercomplex multiplication layer generalize beyond fixed 4D/8D/16D spaces to arbitrary dimensions?
• RQ2 Does PHM maintain or improve performance while significantly reducing parameter counts across NLP tasks?
• RQ3 How does PHM perform in LSTM and Transformer architectures on tasks such as natural language inference and machine translation?
• RQ4 What is the impact of different n values on performance and parameter efficiency across tasks?

Key findings

• PHM-LSTM reduces parameters by up to 88.7% with varying n and generally matches or improves on standard LSTM across five NLI datasets.
• PHM-Transformer achieves substantial parameter reduction (up to 93.4%) with competitive BLEU scores on seven MT datasets; in many cases n=4 or n=2 yields gains over Quaternion/standard Transformers.
• PHM layers offer architectural flexibility, delivering performance gains over baselines on several tasks including text style transfer and subject-verb agreement.
• Increasing n provides more parameter savings but may lead to minor degradations on some datasets (e.g., En-Id) with very large n.
• PHM can learn Hamilton-product-like behavior (n=4) and, with data-driven learning, often outperforms fixed hypercomplex rules.
• In MT, re-scaling hidden size (doubling) with PHM can improve performance on several datasets.