[Paper Review] Learning with Pseudo-Ensembles

Phil Bachman, Ouais Alsharif+1 more|arXiv (Cornell University)|Dec 16, 2014

Tensor decomposition and applications22 references360 citations

TL;DR

This paper introduces pseudo-ensembles—a framework for training robust models by generating a family of perturbed child models from a single parent model via controlled noise processes. It proposes a novel regularizer that enhances model robustness to input and structural noise, achieving state-of-the-art performance in semi-supervised learning and significantly improving a Recursive Neural Tensor Network on sentiment analysis benchmarks beyond the original model.

ABSTRACT

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.

Motivation & Objective

• To formalize the concept of pseudo-ensembles as a unifying framework for methods like dropout and feature noising.
• To develop a regularizer that makes model behavior robust to perturbations in both input and internal structure.
• To extend the applicability of dropout-like regularization to semi-supervised learning, where it traditionally lacks strong theoretical grounding.
• To improve performance of existing models, such as the Recursive Neural Tensor Network, through systematic perturbation of parameters and latent spaces.

Proposed method

• A pseudo-ensemble is defined as a collection of child models generated by perturbing a parent model using a noise process $ p_{\xi} $, where each $ f_{\theta}(x;\xi) $ is a perturbed version of the parent model $ f_{\theta} $.
• The core regularizer minimizes the expected loss over all perturbed child models, formalized as $ \mathbb{E}_{(x,y)\sim p_{xy}} \mathbb{E}_{\xi\sim p_{\xi}} \mathcal{L}(f_{\theta}(x;\xi), y) $.
• Two perturbation mechanisms are applied to the RNTN: (1) subspace sampling, where only half the latent dimensions are used per forward pass, and (2) weight fuzzing, where parameters are perturbed with zero-mean Gaussian noise during training.
• Weight fuzzing is interpreted as implicitly convolving the loss function with a Gaussian, reducing curvature and improving optimization stability.
• The method uses parameter slicing to maintain computational efficiency during training, especially critical for models with $ \mathcal{O}(n^3) $ complexity like RNTNs.
• The framework naturally extends to semi-supervised learning, where it outperforms existing methods on real-world datasets.

Experimental results

Research questions

• RQ1 How can we formally define and unify methods like dropout and feature noising under a single framework of model perturbation?
• RQ2 What is the relationship between pseudo-ensembles and traditional ensemble methods that rely on independent model training?
• RQ3 Can a regularizer based on robustness to model-space perturbations outperform standard dropout in fully-supervised settings?
• RQ4 Does this regularizer generalize effectively to semi-supervised learning, where it has not been widely applied before?
• RQ5 Can the performance of a strong model like the RNTN be significantly improved through structured perturbations in its latent space and parameters?

Key findings

• The proposed regularizer matches the performance of standard dropout in fully-supervised settings, validating its effectiveness.
• The method achieves state-of-the-art performance in semi-supervised learning, outperforming existing approaches on real-world datasets.
• When applied to the Recursive Neural Tensor Network, the pseudo-ensemble approach with both subspace sampling and weight fuzzing achieved 88.9% accuracy on the binary sentiment classification task, surpassing the original RNTN and other baselines.
• Subspace sampling alone improved performance over the original RNTN, and weight fuzzing further enhanced it, with both perturbations contributing additively to gains.
• The combination of both perturbation types pushed the compact RNTN model beyond the performance of the full, original RNTN, demonstrating the power of structured model-space regularization.
• The use of parameter slicing enabled efficient training of larger models, making the method scalable despite the $ \mathcal{O}(n^3) $ complexity of the RNTN.