[Paper Review] Deep Models of Interactions Across Sets

Jason Hartford, Devon R. Graham+2 more|arXiv (Cornell University)|Mar 7, 2018

Topic Modeling25 references52 citations

TL;DR

The paper introduces permutation equivariant neural layers for exchangeable matrices/tensors to model interactions across multiple sets, enabling scalable, inductive matrix/tensor completion and strong cross-domain extrapolation.

ABSTRACT

We use deep learning to model interactions across two or more sets of objects, such as user-movie ratings, protein-drug bindings, or ternary user-item-tag interactions. The canonical representation of such interactions is a matrix (or a higher-dimensional tensor) with an exchangeability property: the encoding's meaning is not changed by permuting rows or columns. We argue that models should hence be Permutation Equivariant (PE): constrained to make the same predictions across such permutations. We present a parameter-sharing scheme and prove that it could not be made any more expressive without violating PE. This scheme yields three benefits. First, we demonstrate state-of-the-art performance on multiple matrix completion benchmarks. Second, our models require a number of parameters independent of the numbers of objects, and thus scale well to large datasets. Third, models can be queried about new objects that were not available at training time, but for which interactions have since been observed. In experiments, our models achieved surprisingly good generalization performance on this matrix extrapolation task, both within domains (e.g., new users and new movies drawn from the same distribution used for training) and even across domains (e.g., predicting music ratings after training on movies).

Motivation & Objective

• Motivate learning interactions across two or more sets (e.g., users-items, proteins-drugs) under exchangeability constraints.
• Introduce a permutation equivariant (PE) parameter-sharing scheme that is maximally expressive without violating PE.
• Develop architectures (exchangeable matrix/tensor layers) that scale with data size and support inductive generalization to unseen entities.
• Demonstrate strong empirical performance on matrix completion benchmarks and cross-domain extrapolation.
• Provide practical considerations for sparse inputs and subsampling in large datasets.

Proposed method

• Define and enforce permutation equivariance for matrix inputs via a tied-parameter layer that reduces to a simple, PE-preserving form (Equation 3).
• Generalize the PE layer to multi-channel inputs and cross-channel interactions (Equation 4).
• Extend the PE framework to higher-dimensional tensors with a corresponding parameter-tying scheme (Theorem 5.1).
• Handle sparsity by adapting pooling terms to only observed entries.
• Propose two training/inference architectures: a self-supervised exchangeable model and a Factorized Exchangeable Autoencoder (FEA).
• Discuss practical regularization (channel dropout) and subsampling strategies for large matrices (uniform and conditional sampling).

Experimental results

Research questions

• RQ1 How can one enforce permutation equivariance in neural models for exchangeable matrix/tensor data without exploding parameter counts?
• RQ2 Can PE-based models achieve competitive or state-of-the-art performance on matrix completion benchmarks in transductive settings while remaining inductive-capable?
• RQ3 Do PE models generalize to unseen users/items (inductive matrix extrapolation) and even cross-domain datasets (e.g., movie ratings to music ratings)?
• RQ4 What is the impact of sparsity and subsampling on the performance of exchangeable PE models, and how can we mitigate it?
• RQ5 How can the PE framework be extended from matrices to higher-order tensors and other exchangeable structures (e.g., graphs via jointly exchangeable matrices)?

Key findings

• The exchangeable matrix layer enforces permutation equivariance with a minimal, highly structured parameter sharing, enabling PE functions.
• The self-supervised exchangeable model achieves state-of-the-art performance on MovieLens-100K in the transductive setting.
• The Factorized Exchangeable Autoencoder (FEA) supports inductive completion and can generalize to unseen users and items, outperforming baselines in inductive tests.
• The approach yields competitive results on larger datasets when memory constraints necessitate subsampling, with conditional sampling helping recover some performance.
• The method demonstrates strong cross-domain extrapolation, performing competitively when trained on movie ratings and tested on music ratings (after appropriate binning/rescaling).
• The framework naturally extends to higher-order tensors and connects to graph-convolutional architectures as a special case of parameter sharing.