[论文解读] Deep Sets

Manzil Zaheer, Satwik Kottur+4 more|arXiv (Cornell University)|Mar 10, 2017

Anomaly Detection Techniques and Applications被引用 76

一句话总结

Deep Sets 引入了一种处理置换不变集合输入的原理性架构，证明通用形式 f(X)=ρ(Σφ(x)) 并推导神经网络的置换等变层，其应用广泛，从统计到点云分类以及集合扩展。

ABSTRACT

We study the problem of designing models for machine learning tasks defined on \emph{sets}. In contrast to traditional approach of operating on fixed dimensional vectors, we consider objective functions defined on sets that are invariant to permutations. Such problems are widespread, ranging from estimation of population statistics \cite{poczos13aistats}, to anomaly detection in piezometer data of embankment dams \cite{Jung15Exploration}, to cosmology \cite{Ntampaka16Dynamical,Ravanbakhsh16ICML1}. Our main theorem characterizes the permutation invariant functions and provides a family of functions to which any permutation invariant objective function must belong. This family of functions has a special structure which enables us to design a deep network architecture that can operate on sets and which can be deployed on a variety of scenarios including both unsupervised and supervised learning tasks. We also derive the necessary and sufficient conditions for permutation equivariance in deep models. We demonstrate the applicability of our method on population statistic estimation, point cloud classification, set expansion, and outlier detection.

研究动机与目标

• Motivate learning tasks where inputs are sets rather than fixed-size vectors and invariance to element order is required.
• Characterize the structure of permutation-invariant functions and identify a universal form.
• Develop DeepSets architecture that operates on sets of varying sizes and supports conditioning on meta-information.
• Derive necessary and sufficient conditions for permutation equivariance in neural layers.
• Demonstrate the approach across diverse supervised and unsupervised tasks with empirical results.

提出的方法

• Prove that a permutation-invariant set function consists of a φ-transform of each element followed by a sum and a ρ-transform, i.e., f(X)=ρ(Σx∈Xφ(x)).
• Extend the architecture to allow conditioning on auxiliary information via φ(x|z) and ρ.
• Derive the conditions for permutation-equivariant neural network layers, showing that a layer is equivariant iff its weight matrix has the form Θ=λI+γ(11^T).
• Develop DeepSets by stacking permutation-equivariant layers and combining them with set pooling (sum or max) to handle sets of varying size.
• Demonstrate supervision, semi-supervision, and conditioning variants to handle diverse tasks.
• Apply the framework to population statistics estimation, point-cloud classification, set expansion, and outlier detection.

实验结果

研究问题

• RQ1 What is the universal structure of permutation-invariant functions over sets?
• RQ2 Can a neural network architecture be built to respect permutation invariance and handle variable set sizes?
• RQ3 What are the necessary and sufficient conditions for permutation equivariance in neural layers?
• RQ4 How can conditioning on additional information be integrated into set-based models?
• RQ5 How does DeepSets perform across tasks like statistics estimation, point-cloud classification, set expansion, and anomaly detection?

主要发现

• A permutation-invariant set function can be decomposed as f(X)=ρ(Σφ(x)) for countable X, with conjectured extension to uncountable X.
• A neural network layer is permutation-equivariant iff its weight matrix has the form Θ=λI+γ(11^T).
• DeepSets achieves competitive or superior performance across diverse tasks including population statistics estimation, point-cloud classification, set expansion, and outlier detection.
• The framework accommodates conditioning on meta-information, enabling flexible fusion of multiple data sources.
• Experimentally, DeepSets demonstrates strong generalization and applicability without task-specific architectures.