[Paper Review] On Controllable Sparse Alternatives to Softmax

Anirban Laha, Saneem Chemmengath+4 more|arXiv (Cornell University)|Oct 29, 2018

Domain Adaptation and Few-Shot Learning20 citations

TL;DR

This paper proposes a unified framework for sparse probability mapping functions that generalizes softmax, sparsemax, and other variants, enabling explicit control over sparsity through two new formulations—sparsegen-lin and sparsehourglass. The framework supports closed-form solutions and backpropagation, demonstrating improved performance in multilabel classification and sequence-to-sequence tasks like machine translation and summarization.

ABSTRACT

Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilabel classification, attention mechanisms etc. For this, several probability mapping functions have been proposed and employed in literature such as softmax, sum-normalization, spherical softmax, and sparsemax, but there is very little understanding in terms how they relate with each other. Further, none of the above formulations offer an explicit control over the degree of sparsity. To address this, we develop a unified framework that encompasses all these formulations as special cases. This framework ensures simple closed-form solutions and existence of sub-gradients suitable for learning via backpropagation. Within this framework, we propose two novel sparse formulations, sparsegen-lin and sparsehourglass, that seek to provide a control over the degree of desired sparsity. We further develop novel convex loss functions that help induce the behavior of aforementioned formulations in the multilabel classification setting, showing improved performance. We also demonstrate empirically that the proposed formulations, when used to compute attention weights, achieve better or comparable performance on standard seq2seq tasks like neural machine translation and abstractive summarization.

Motivation & Objective

• To unify existing probability mapping functions like softmax, sparsemax, and sum-normalization under a single theoretical framework.
• To address the lack of explicit control over sparsity in existing sparse mappings.
• To develop novel convex loss functions tailored for multilabel classification using the proposed formulations.
• To enable end-to-end learning with subgradient support via closed-form solutions.
• To evaluate the proposed methods in attention mechanisms for sequence-to-sequence tasks.

Proposed method

• The framework generalizes existing mappings by introducing a parameterized family of functions that subsume softmax, sparsemax, and sum-normalization as special cases.
• The proposed sparsegen-lin formulation uses linear constraints to control sparsity, enabling explicit tuning of the number of non-zero outputs.
• The sparsehourglass formulation introduces a two-stage optimization process that first identifies active components and then normalizes them, allowing for flexible sparsity control.
• The framework ensures existence of subgradients through convex optimization, enabling backpropagation in deep learning models.
• Novel convex loss functions are designed to induce desired sparsity behavior in multilabel classification.
• The method is evaluated in attention mechanisms by replacing standard softmax with the proposed sparse alternatives in seq2seq models.

Experimental results

Research questions

• RQ1 How can existing sparse probability mappings like softmax and sparsemax be unified under a single theoretical framework?
• RQ2 Can explicit control over sparsity be achieved in probability mapping functions without sacrificing differentiability or closed-form solutions?
• RQ3 How do the proposed formulations—sparsegen-lin and sparsehourglass—perform in multilabel classification compared to existing methods?
• RQ4 What impact does sparsity control have on attention mechanisms in sequence-to-sequence models like neural machine translation and abstractive summarization?
• RQ5 Can novel convex loss functions be designed to effectively induce the desired sparsity behavior in multilabel learning?

Key findings

• The proposed framework generalizes existing mappings such as softmax, sparsemax, and sum-normalization as special cases, providing a unified view.
• sparsegen-lin and sparsehourglass enable explicit control over the degree of sparsity through tunable parameters.
• The formulations support closed-form solutions and subgradients, enabling end-to-end training via backpropagation.
• In multilabel classification, the novel convex loss functions lead to improved performance compared to standard approaches.
• In sequence-to-sequence tasks, the proposed sparse attention mechanisms achieve performance on par with or better than standard softmax.