[Paper Review] Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

Ryan L. Murphy, Balasubramaniam Srinivasan+2 more|arXiv (Cornell University)|Nov 5, 2018

Machine Learning in Bioinformatics46 references65 citations

TL;DR

The paper introduces Janossy pooling to learn permutation-invariant functions for variable-size inputs by averaging permutation-sensitive function outputs across all input orderings, and proposes tractable approximations and learning algorithms.

ABSTRACT

We consider a simple and overarching representation for permutation-invariant functions of sequences (or multiset functions). Our approach, which we call Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence. This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and flexible permutation-invariant functions. If carried out naively, Janossy pooling can be computationally prohibitive. To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with $k$-order interactions, and stochastic optimization algorithms with random permutations. Our framework unifies a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions. We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.

Motivation & Objective

• Develop a flexible, learnable pooling operator that is invariant to input ordering and supports variable-length inputs.
• Leverage permutation-sensitive functions to construct powerful permutation-invariant neural networks.
• Provide tractable strategies (canonical orderings, k-ary dependencies, permutation sampling) and unify existing methods within the Janossy pooling framework.
• Analyze connections to existing works like DeepSets and de Finetti exchangeability, and study training dynamics.
• Demonstrate empirically that Janossy pooling improves over state-of-the-art baselines in relevant tasks.

Proposed method

• Define Janossy pooling as an average over outputs of a permutation-sensitive function applied to all input permutations.
• Introduce a three-pronged tractability approach: canonical input orderings, k-ary dependencies, and permutation sampling (pi-SGD).
• Show that k-ary Janossy pooling reduces computation to a subset of permutations.
• Provide analysis linking DeepSets to 1-ary Janossy pooling and discuss trade-offs in expressivity vs. computation.
• Propose pi-SGD to train permutation-sensitive models by sampling random permutations.
• Discuss inference implications and variance-reduction ideas for trained Janossy models.

Experimental results

Research questions

• RQ1 How can permutation-invariant functions be learned flexibly for variable-size inputs?
• RQ2 What tractable approximations to the full Janossy pooling can preserve expressivity while reducing computation?
• RQ3 How do canonical orderings, k-ary dependencies, and permutation sampling compare in practice for learning permutation-invariant models?
• RQ4 What theoretical connections exist between Janossy pooling, DeepSets, and finite de Finetti exchangeability?
• RQ5 How does pi-SGD behave in terms of convergence and inference when using permutation-based training?

Key findings

• Janossy pooling provides a unifying framework to construct permutation-invariant networks from permutation-sensitive components.
• Three tractability strategies are proposed: canonical orderings, k-ary dependencies, and permutation sampling, with trade-offs between expressivity and cost.
• k-ary Janossy pooling reduces the combinatorial burden by summing over a reduced set of permutations.
• pi-SGD offers a principled stochastic optimization method for training Janossy-based models by sampling random input permutations.
• The approach yields improved performance over state-of-the-art baselines in experiments (as claimed).
• A probabilistic link is established between Janossy pooling and finite de Finetti exchangeability.