[Paper Review] Deep Combinatorial Aggregation

Yuesong Shen, Daniel Cremers|arXiv (Cornell University)|Oct 12, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper introduces Deep Combinatorial Aggregation (DCA), a combinatorial generalization of deep ensemble that enhances uncertainty-aware learning by aggregating diverse model proposals through hierarchical network component combinations. DCA achieves state-of-the-art performance in both predictive accuracy and uncertainty estimation, with fine-grain DCA enabling a new weight averaging method (DCWA) that matches SWA without custom training schedules or BatchNorm adjustments.

ABSTRACT

Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue. This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks. In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA). DCA creates multiple instances of network components and aggregates their combinations to produce diversified model proposals and predictions. DCA components can be defined at different levels of granularity. And we discovered that coarse-grain DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation. For fine-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training. It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers. Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA. We experiment on in-domain, distributional shift, and out-of-distribution image classification tasks, and empirically confirm the effectiveness of DCWA and DCA approaches.

Motivation & Objective

• To address the poor uncertainty estimation of neural networks in safety-critical and active learning applications.
• To generalize deep ensemble by exploring combinatorial aggregation of network components at multiple granularities.
• To develop a new weight averaging technique (DCWA) that improves generalization without requiring custom training schedules or BatchNorm adjustments.
• To introduce a consistency enforcing loss that enhances predictive consistency and uncertainty calibration in DCA models.
• To empirically validate DCA across in-domain, distributional shift, and out-of-distribution image classification benchmarks.

Proposed method

• DCA constructs multiple model proposals by combining different instances of network components (e.g., layers, blocks, or entire models) in a combinatorial fashion.
• For fine-grain DCA, the method introduces Deep Combinatorial Weight Averaging (DCWA), which computes an average of model weights across component instances without requiring iterative training or learning rate scheduling.
• The consistency enforcing loss is applied during DCA training to improve agreement among model proposals, enhancing both accuracy and uncertainty calibration.
• DCA is trained using a modified backpropagation scheme that accumulates gradients over multiple randomly sampled component proposals per minibatch to ensure joint optimization.
• The method supports multiple granularities: layerwise, trunkwise, and modelwise DCA, each enabling different trade-offs between diversity and performance.
• DCWA is trained end-to-end with standard cross-entropy or negative log-likelihood loss, and can be deployed as a single model without ensemble inference.

Experimental results

Research questions

• RQ1 Can a combinatorial generalization of deep ensemble improve uncertainty estimation beyond standard deep ensemble?
• RQ2 Does fine-grain DCA with DCWA yield performance comparable to stochastic weight averaging (SWA) without requiring custom training schedules?
• RQ3 How does the consistency enforcing loss affect the predictive and uncertainty calibration performance of DCA models?
• RQ4 Can coarse-grain DCA variants (e.g., modelwise DCA) outperform deep ensemble across in-domain, distributional shift, and out-of-distribution scenarios?
• RQ5 What is the impact of the number of DCA component instances on model performance and computational cost?

Key findings

• Modelwise DCA achieved the highest accuracy (94.95% ± 0.0008) and lowest NLL (0.1601 ± 0.0006) on CIFAR-10, outperforming all baselines including deep ensemble and SWA.
• DCWA with negative log-likelihood loss achieved 93.14% ± 0.0017 accuracy and 0.0365 ± 0.0014 ECE, showing strong uncertainty calibration.
• The consistency enforcing loss improved the uncertainty estimation of modelwise DCA, reducing ECE from 0.0107 ± 0.0004 to 0.0084 ± 0.0009 when trained with the loss.
• Trunkwise DCA produced more diverse predictions than deep ensemble while maintaining competitive performance, with 94.54% ± 0.0008 accuracy and 0.0080 ± 0.0011 ECE.
• Increasing the number of DCA component instances (from 2 to 5) led to consistent improvements in accuracy and NLL, demonstrating a favorable trade-off between computational cost and performance.
• DCWA matched the performance of SWA but required no learning rate scheduling or BatchNorm adaptation, making it easier to implement and deploy.