[Paper Review] Mind the Nuisance: Gaussian Process Classification using Privileged Noise

Daniel Hernández-Lobato, Viktoriia Sharmanska+3 more|arXiv (Cornell University)|Jul 1, 2014

Gaussian Processes and Bayesian Inference30 references19 citations

TL;DR

This paper proposes GPC+, a novel Gaussian Process Classifier that treats privileged information as data-dependent noise in the latent function, enabling the model to modulate the sigmoid likelihood slope to reflect confidence in training examples. Experiments show GPC+ outperforms standard GPC and SVM+ on public datasets, including with deep features and semantic attributes as privileged inputs, achieving state-of-the-art accuracy with interpretable uncertainty modeling.

ABSTRACT

The learning with privileged information setting has recently attracted a lot of attention within the machine learning community, as it allows the integration of additional knowledge into the training process of a classifier, even when this comes in the form of a data modality that is not available at test time. Here, we show that privileged information can naturally be treated as noise in the latent function of a Gaussian Process classifier (GPC). That is, in contrast to the standard GPC setting, the latent function is not just a nuisance but a feature: it becomes a natural measure of confidence about the training data by modulating the slope of the GPC sigmoid likelihood function. Extensive experiments on public datasets show that the proposed GPC method using privileged noise, called GPC+, improves over a standard GPC without privileged knowledge, and also over the current state-of-the-art SVM-based method, SVM+. Moreover, we show that advanced neural networks and deep learning methods can be compressed as privileged information.

Motivation & Objective

• To integrate privileged information—available only at training time—into Gaussian Process Classification in a principled, interpretable way.
• To model classifier confidence via data-dependent noise modulation in the latent function, improving generalization.
• To extend Bayesian classification to heteroscedastic noise in GPC, a previously unexplored setting.
• To demonstrate that modern deep neural network features (e.g., DeCAF) and semantic attributes can serve as effective privileged information.
• To develop a variational inference method for approximate posterior computation in this novel heteroscedastic GPC framework.

Proposed method

• Privileged information is modeled as a noise term in the latent function of a Gaussian Process classifier, which modulates the slope of the sigmoid likelihood.
• The noise is integrated out during training, ensuring privileged data is not needed at test time.
• A variational inference approach based on expectation propagation is used to approximate the intractable marginal likelihood and predictive distribution.
• The model learns a data-dependent noise level per training example, where high noise indicates low confidence and low noise indicates high confidence.
• The method is applied to both standard and deep features (e.g., DeCAF) as privileged inputs, with PCA and standardization for dimensionality reduction.
• The resulting GPC+ model interprets the sigmoid slope as a measure of training example reliability, with steeper slopes indicating higher confidence in the label.

Experimental results

Research questions

• RQ1 Can privileged information be effectively modeled as noise in a Gaussian Process classifier to improve performance?
• RQ2 How does modeling privileged data as noise affect the interpretability and confidence estimation of the classifier?
• RQ3 Does the proposed GPC+ method outperform standard GPC and SVM+ in terms of classification accuracy on benchmark datasets?
• RQ4 Can deep neural network representations (e.g., DeCAF) and semantic attributes serve as effective privileged information in a Bayesian classification framework?
• RQ5 Is the proposed variational inference method for heteroscedastic GPC classification both effective and scalable?

Key findings

• GPC+ significantly outperforms standard GPC and SVM+ on the AwA dataset when using DeCAF features as privileged information, with statistical evidence of superiority (p < 10%).
• On the AwA dataset with semantic attributes as privileged data, GPC+ achieves the best average rank among the four methods, though the performance difference with GPC is not statistically significant.
• The model successfully uses deep features (4096D DeCAF) and semantic attributes (85D) as privileged inputs, demonstrating the method's flexibility across data modalities.
• The slope of the sigmoid likelihood function in GPC+ provides an interpretable measure of confidence: steeper slopes correspond to easier, more confidently classified examples.
• Experiments show that GPC+ improves generalization by assigning higher importance to training examples that are easily classified via privileged data.
• The method is the first to apply heteroscedastic noise modeling in Gaussian Process classification, enabling data-dependent uncertainty quantification in a Bayesian framework.