[Paper Review] Déjà Vu: an empirical evaluation of the memorization properties of ConvNets

Alexandre Sablayrolles, Matthijs Douze+2 more|arXiv (Cornell University)|Sep 17, 2018

Opportunistic and Delay-Tolerant Networks36 references10 citations

TL;DR

This paper investigates memorization in modern ConvNets by proposing membership inference attacks that detect whether specific images were used in training, even when only intermediate layers are accessible. It demonstrates that models like ResNet-101 and VGG-16 can memorize training data, with attack accuracy dropping significantly under data augmentation, and shows that lower layers retain detectable memorization signals despite fine-tuning or layer truncation.

ABSTRACT

Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting. This paper considers the related question of "membership inference", where the goal is to determine if an image was used during training. We consider it under three complementary angles. We show how to detect which dataset was used to train a model, and in particular whether some validation images were used at train time. We then analyze explicit memorization and extend classical random label experiments to the problem of learning a model that predicts if an image belongs to an arbitrary set. Finally, we propose a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples. To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

Motivation & Objective

• To investigate whether modern ConvNets like ResNet-101 and VGG-16 memorize training images, particularly in the context of privacy and data leakage.
• To develop and evaluate membership inference attacks that detect if a specific image or dataset subset was used during training, even when the final layers are unavailable.
• To analyze the impact of data augmentation (e.g., flip, crop, resize) on memorization and attack performance, assessing its role in privacy preservation.
• To extend membership inference to transferred and fine-tuned networks by leveraging intermediate feature representations instead of final-layer outputs.
• To provide empirical evidence on the capacity of ConvNets to explicitly memorize arbitrary image sets, using controlled experiments with random labels and synthetic datasets.

Proposed method

• Proposes a binary classifier as a drop-in replacement for the final layer to perform membership inference, using the model’s confidence score as a signal.
• Employs two attack strategies: the Bayes attack (based on class-conditional probability density) and the Model Accuracy Threshold (MAT) attack (using loss distribution differences between training and held-out sets).
• Uses the cumulative distribution function (CDF) of losses on training and held-out sets to compute the optimal threshold τ, maximizing attack accuracy via equation: $ \text{accuracy} = \frac{1}{2} + \frac{1}{2}(F_{\text{train}}(\tau) - F_{\text{heldout}}(\tau)) $.
• Conducts experiments on ImageNet and YFCC-100M using VGG-16 and ResNet-101 with varying data augmentation levels (none, flip+crop±5, flip+crop).
• For partial-layers attacks, retrains the missing top layers using public data and applies the Bayes or MAT attack on the remaining feature representations.
• Evaluates attack performance on different network layers (e.g., last block, 4th stage of ResNet) to assess memorization retention in lower layers.

Experimental results

Research questions

• RQ1 To what extent can modern ConvNets like ResNet-101 and VGG-16 memorize specific training images, even with strong data augmentation?
• RQ2 Can membership inference attacks detect whether a model was trained on a subset of its validation set, even when the training set is large and augmented?
• RQ3 Can membership inference be performed effectively when only intermediate layers are accessible, such as in fine-tuned or transferred models?
• RQ4 How does data augmentation (e.g., random cropping, flipping) affect the memorization capacity and the success rate of membership inference attacks?
• RQ5 What is the upper bound on the number of images a ConvNet can explicitly memorize, and how does this relate to model capacity and optimization dynamics?

Key findings

• Without data augmentation, membership inference attacks on the final layer of ResNet-101 and VGG-16 achieve 90.4% and 90.8% accuracy, respectively, indicating strong memorization.
• With full data augmentation (flip+crop±5), attack accuracy drops to 77.4% (ResNet-101) and 79.5% (VGG-16), showing that data augmentation reduces memorization leakage.
• Even when only the last block of features is available (e.g., after fine-tuning), attacks achieve 53.1% (ResNet-101) and 51.7% (VGG-16) accuracy, indicating that intermediate layers still carry detectable memorization signals.
• For models trained with minimal augmentation, membership inference on intermediate layers (e.g., 4th stage of ResNet) achieves 65.7% accuracy, demonstrating that lower layers retain significant memorization.
• The MAT attack outperforms random guessing and is robust across different architectures and augmentation levels, with accuracy consistently above 50% even under strong data augmentation.
• The study provides empirical evidence that data augmentation is an effective privacy-preserving technique, as it significantly reduces the success of membership inference attacks on large-scale datasets like ImageNet.