[Paper Review] Tighter risk certificates for neural networks

María Pérez‐Ortiz, Omar Rivasplata+2 more|Research Explorer (The University of Manchester)|Jul 25, 2020

Adversarial Robustness in Machine LearningComputer Science63 references32 citations

TL;DR

This paper empirically studies training probabilistic neural networks using training objectives derived from tight PAC-Bayes risk bounds, proposing two new objectives and comparing with a classic bound to obtain non-vacuous, tighter risk certificates and competitive test errors.

ABSTRACT

This paper presents an empirical study regarding training probabilistic neural networks using training objectives derived from PAC-Bayes bounds. In the context of probabilistic neural networks, the output of training is a probability distribution over network weights. We present two training objectives, used here for the first time in connection with training neural networks. These two training objectives are derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned using the different training objectives. We compute risk certificates for the learnt predictors, based on part of the data used to learn the predictors. We further experiment with different types of priors on the weights (both data-free and data-dependent priors) and neural network architectures. Our experiments on MNIST and CIFAR-10 show that our training methods produce competitive test set errors and non-vacuous risk bounds with much tighter values than previous results in the literature, showing promise not only to guide the learning algorithm through bounding the risk but also for model selection. These observations suggest that the methods studied here might be good candidates for self-certified learning, in the sense of using the whole data set for learning a predictor and certifying its risk on any unseen data (from the same distribution as the training data) potentially without the need for holding out test data.

Motivation & Objective

• Investigate training probabilistic neural networks using PAC-Bayes bounds as training objectives.
• Introduce two new PAC-Bayes-based training objectives derived from tight bounds.
• Compare new objectives with a classic PAC-Bayes objective to assess predictor quality and risk certificates.
• Demonstrate the ability to compute tight, non-vacuous risk certificates on MNIST and CIFAR-10.

Proposed method

• Define neural networks as distributions over weights and train via stochastic gradient descent.
• Develop two new objectives: f_quad derived from PAC-Bayes-quadratic bound and f_lambda from PAC-Bayes-lambda bound.
• Re-implement the classical PAC-Bayes objective (f_classic) for comparison.
• Compute risk certificates for learned predictors using part of the training data.
• Experiment with data-free and data-dependent priors and various architectures.
• Relate PAC-Bayes with Backprop to Bayes-by-Backprop and contrast training strategies.

Experimental results

Research questions

• RQ1 Can PAC-Bayes-based training objectives yield competitive test errors while providing non-vacuous, tight risk certificates?
• RQ2 Do the proposed f_quad and f_lambda objectives produce tighter certificates than the classic PAC-Bayes objective?
• RQ3 How do data-dependent vs data-free priors affect risk certificates and predictive performance?
• RQ4 Do the methods enable self-certified learning by using all data for learning and certifying risk on unseen data?
• RQ5 What is the impact of network architecture (fully connected vs convolutional) on certificate tightness and accuracy?

Key findings

• The proposed PBB training objectives achieve competitive test set errors compared to existing methods.
• The risk certificates produced by the new objectives are tighter than those from the classic bound.
• The approach demonstrates non-vacuous risk bounds for learned neural networks on MNIST and CIFAR-10.
• Re-implementation of the classic objective can also yield improved results, suggesting training strategy contributes to gains.
• Data-dependent priors and varying architectures influence the tightness of certificates and predictive performance.
• The results support self-certified learning by combining predictor learning with risk certification on unseen data.