[Paper Review] Summary Statistics for Partitionings and Feature Allocations

Işık Barış Fidaner, Ali Taylan Cemgil|arXiv (Cornell University)|Oct 1, 2013

Bayesian Methods and Mixture Models18 references3 citations

TL;DR

This paper introduces entropy agglomeration (EA), a novel method for summarizing and visualizing sample sets of partitionings and feature allocations from infinite mixture models. By defining element-based entropy and using cumulative statistics, EA generates interpretable dendrograms that reveal cluster structures, segmentation patterns, and uncertainty in posterior samples, demonstrating strong performance on synthetic, gene expression, and intergovernmental organization datasets.

ABSTRACT

Infinite mixture models are commonly used for clustering. One can sample from the posterior of mixture assignments by Monte Carlo methods or find its maximum a posteriori solution by optimization. However, in some problems the posterior is diffuse and it is hard to interpret the sampled partitionings. In this paper, we introduce novel statistics based on block sizes for representing sample sets of partitionings and feature allocations. We develop an element-based definition of entropy to quantify segmentation among their elements. Then we propose a simple algorithm called entropy agglomeration (EA) to summarize and visualize this information. Experiments on various infinite mixture posteriors as well as a feature allocation dataset demonstrate that the proposed statistics are useful in practice.

Motivation & Objective

• To address the challenge of summarizing diffuse posterior distributions over partitionings and feature allocations in infinite mixture models.
• To develop a systematic way to represent sample sets of partitionings and feature allocations using cumulative statistics and occurrence matrices.
• To define per-element information and entropy sequences for quantifying segmentation and uncertainty in cluster structures.
• To create a simple yet effective algorithm—entropy agglomeration (EA)—that visualizes key structural patterns in sample sets.
• To demonstrate the utility of the proposed statistics on diverse real and synthetic datasets, including gene expression and intergovernmental organization memberships.

Proposed method

• Proposes a new definition of entropy based on per-element information to quantify segmentation among elements in partitionings and feature allocations.
• Introduces cumulative statistics and cumulative occurrence distribution matrices to systematically represent sample sets of partitionings and feature allocations.
• Develops the entropy agglomeration (EA) algorithm to select and visualize a small subset of entropy sequences that best represent the structure of the sample set.
• Uses pairwise occurrence probabilities ordered by the EA dendrogram to visualize co-occurrence patterns and cluster hierarchy.
• Applies the method to both posterior samples from infinite mixture models and direct feature allocation datasets, such as IGO memberships.
• Employs a hierarchical agglomeration process guided by entropy sequences to build dendrograms that reflect the underlying structure of the data.

Experimental results

Research questions

• RQ1 How can we effectively summarize and visualize sample sets of partitionings and feature allocations when the posterior distribution is diffuse?
• RQ2 What is a meaningful way to define entropy for partitionings and feature allocations that captures per-element information and segmentation?
• RQ3 Can cumulative statistics and entropy sequences be used to construct a hierarchical representation of cluster structures in sample sets?
• RQ4 How well does the entropy agglomeration (EA) algorithm perform in revealing true underlying structures in diverse datasets?
• RQ5 Can the proposed method be applied directly to real-world feature allocation data, such as intergovernmental organization memberships?

Key findings

• The entropy agglomeration (EA) algorithm successfully reveals the underlying cluster structure in synthetic data with three clearly separated clusters, distinguishing inner and outer elements based on their dispersion.
• On the Iris flower dataset, EA correctly groups all 50 samples of each species into single leaves, reflecting their strong separability and low uncertainty in the posterior.
• For the Galactose gene expression dataset, EA identifies distinct subgroups such as 70 ribosomal protein (RP) genes and 12 hexose transport (HX) genes as individual leaves, and reveals a hierarchical structure with an outer group of 19 genes and an inner cluster of 68 genes.
• In the IGO dataset, EA directly applied to feature allocations (IGO-year memberships) recovers a meaningful continental ordering: Europe, America-Australia-NZ, Asia, Africa, and Middle East, demonstrating its utility on non-posterior data.
• The EA dendrograms, when paired with ordered pairwise occurrence matrices, effectively visualize co-occurrence patterns and structural uncertainty, with shades of gray indicating varying degrees of cluster cohesion.
• The method is simple to implement and does not require deep prior knowledge, yet it provides meaningful, interpretable summaries of complex posterior sample sets in diverse domains.