[Paper Review] Bayesian Image Classification with Deep Convolutional Gaussian Processes

Vincent Dutordoir, Mark van der Wilk+2 more|arXiv (Cornell University)|Feb 15, 2019

Gaussian Processes and Bayesian Inference27 references4 citations

TL;DR

This paper introduces the Translation Insensitive Convolutional Kernel (TICK) for deep convolutional Gaussian processes, enabling improved uncertainty calibration and accuracy in image classification by relaxing strict translation invariance. The method achieves state-of-the-art performance on MNIST and CIFAR-10, outperforming both standard convolutional GPs and dropout-based Bayesian deep learning in both accuracy and marginal likelihood-based hyperparameter tuning.

ABSTRACT

In decision-making systems, it is important to have classifiers that have calibrated uncertainties, with an optimisation objective that can be used for automated model selection and training. Gaussian processes (GPs) provide uncertainty estimates and a marginal likelihood objective, but their weak inductive biases lead to inferior accuracy. This has limited their applicability in certain tasks (e.g. image classification). We propose a translation-insensitive convolutional kernel, which relaxes the translation invariance constraint imposed by previous convolutional GPs. We show how we can use the marginal likelihood to learn the degree of insensitivity. We also reformulate GP image-to-image convolutional mappings as multi-output GPs, leading to deep convolutional GPs. We show experimentally that our new kernel improves performance in both single-layer and deep models. We also demonstrate that our fully Bayesian approach improves on dropout-based Bayesian deep learning methods in terms of uncertainty and marginal likelihood estimates.

Motivation & Objective

• Address the limitation of existing convolutional Gaussian processes that enforce strict translation invariance, which restricts model flexibility and performance on image data.
• Improve uncertainty quantification and model calibration in Bayesian image classification by incorporating more flexible inductive biases.
• Enable automatic hyperparameter selection through the use of marginal likelihood in a fully Bayesian framework, overcoming a key limitation of dropout-based Bayesian deep learning.
• Reformulate deep convolutional GP layers as correlated multi-output Gaussian processes to support efficient, scalable training and inference.
• Demonstrate that better inductive biases in the prior lead to superior posterior approximations and model performance, even when Bayesian inference is applied.

Proposed method

• Propose the Translation Insensitive Convolutional Kernel (TICK), which relaxes the requirement that identical patches produce identical outputs, allowing for more flexible feature learning.
• Reformulate image-to-image convolutional mappings in deep GPs as multi-output Gaussian processes, enabling efficient inference using a general-purpose multi-output GP framework.
• Use stochastic variational inference with inducing points to scale the model to large datasets while maintaining computational efficiency.
• Leverage the marginal likelihood as an objective for hyperparameter optimization, enabling automatic model selection and regularization.
• Implement the model within a reusable, open-source multi-output GP framework (van der Wilk et al., 2020), supporting modular and extensible code.
• Apply the model to both single-layer and deep architectures, demonstrating consistent improvements over standard convolutional GPs and Bayesian deep learning baselines.

Experimental results

Research questions

• RQ1 Can relaxing translation invariance in convolutional Gaussian processes lead to improved performance on image classification tasks?
• RQ2 Does the proposed multi-output GP formulation enable efficient and scalable training of deep convolutional GPs?
• RQ3 Can the marginal likelihood in the proposed framework be used effectively for automatic hyperparameter tuning, unlike in dropout-based Bayesian deep learning?
• RQ4 How does the uncertainty calibration of the proposed method compare to that of dropout-based Bayesian neural networks on standard image benchmarks?
• RQ5 To what extent do improved inductive biases in the prior translate into better posterior approximations and model performance in image classification?

Key findings

• The Deep TICK-GP model achieves 99.33% top-1 accuracy on MNIST, outperforming a standard deep GP with 98% accuracy and a non-convolutional deep GP with 98% accuracy.
• On CIFAR-10, the Deep TICK-GP achieves 74.41% top-1 accuracy, significantly outperforming the non-convolutional deep GP baseline at 47.26%.
• The model reduces top-1 error to 0.64% on MNIST with 3 layers, compared to 0.93% for standard convolutional GPs, demonstrating improved accuracy.
• The model achieves lower NLL (0.02) and negative ELBO (4.19 × 10³) on MNIST with 3 layers, indicating better uncertainty calibration and model fit compared to baselines.
• The marginal likelihood estimates in the proposed method are usable for hyperparameter selection, unlike in dropout-based Bayesian deep learning, where such estimates are unreliable.
• The TICK kernel consistently outperforms standard convolutional GPs across all depth settings on both MNIST and CIFAR-10, with improvements in both accuracy and uncertainty quantification.