[Paper Review] On the relationship between class selectivity, dimensionality, and robustness

Matthew L. Leavitt, Ari S. Morcos|arXiv (Cornell University)|Jul 8, 2020

Imbalanced Data Classification Techniques26 references4 citations

TL;DR

This paper investigates the causal relationship between class selectivity—how selectively neurons respond to specific classes—and robustness in deep neural networks. Using a class selectivity regularizer, the authors show that lower selectivity improves robustness to naturalistic corruptions (e.g., blur, brightness) but increases vulnerability to gradient-based adversarial attacks, with increased representational dimensionality of activation changes explaining the adversarial vulnerability.

ABSTRACT

While the relative trade-offs between sparse and distributed representations in deep neural networks (DNNs) are well-studied, less is known about how these trade-offs apply to representations of semantically-meaningful information. Class selectivity, the variability of a unit's responses across data classes or dimensions, is one way of quantifying the sparsity of semantic representations. Given recent evidence showing that class selectivity can impair generalization, we sought to investigate whether it also confers robustness (or vulnerability) to perturbations of input data. We found that mean class selectivity predicts vulnerability to naturalistic corruptions; networks regularized to have lower levels of class selectivity are more robust to corruption, while networks with higher class selectivity are more vulnerable to corruption, as measured using Tiny ImageNetC and CIFAR10C. In contrast, we found that class selectivity increases robustness to multiple types of gradient-based adversarial attacks. To examine this difference, we studied the dimensionality of the change in the representation due to perturbation, finding that decreasing class selectivity increases the dimensionality of this change for both corruption types, but with a notably larger increase for adversarial attacks. These results demonstrate the causal relationship between selectivity and robustness and provide new insights into the mechanisms of this relationship.

Motivation & Objective

• To investigate how class selectivity in deep neural network representations affects robustness to input perturbations.
• To examine whether sparse (high-selectivity) or distributed (low-selectivity) representations confer better robustness under different perturbation types.
• To determine whether class selectivity mediates a trade-off between average-case (corruption) and worst-case (adversarial) robustness.
• To analyze the role of representational dimensionality in mediating the relationship between selectivity and robustness.

Proposed method

• A class selectivity regularizer was applied during training to control the level of selectivity in individual neurons across data classes.
• Class selectivity was quantified using a normalized index based on the difference between maximum and non-maximum class-conditional mean activations.
• Networks were trained on Tiny ImageNet and CIFAR10 with varying regularization strength (α), where negative α reduced selectivity and positive α increased it.
• Robustness was evaluated using Tiny ImageNetC and CIFAR10C for naturalistic corruptions, and FGSM and PGD for adversarial attacks.
• Representational dimensionality was measured via PCA, counting dimensions needed to explain 90% of variance in activation changes due to perturbations.
• The input-output Jacobian was used to assess network stability under adversarial perturbations.

Experimental results

Research questions

• RQ1 Does reducing class selectivity improve robustness to naturalistic image corruptions?
• RQ2 Does reducing class selectivity increase vulnerability to gradient-based adversarial attacks?
• RQ3 How does the dimensionality of activation changes due to perturbations relate to robustness?
• RQ4 Is there a causal link between class selectivity and robustness under different perturbation types?
• RQ5 Does the dimensionality of representation changes explain the observed trade-off between corruption and adversarial robustness?

Key findings

• Lower class selectivity significantly improved robustness to naturalistic corruptions on both Tiny ImageNetC and CIFAR10C, with mean test accuracy increasing as selectivity decreased.
• Reducing class selectivity increased vulnerability to FGSM and PGD adversarial attacks, with test accuracy dropping more sharply for low-selectivity models.
• The dimensionality of activation changes due to adversarial perturbations increased substantially in low-selectivity networks, especially in early layers.
• For naturalistic corruptions, the dimensionality of activation changes also increased with lower selectivity, but to a lesser extent than for adversarial attacks.
• The increase in dimensionality of activation changes under adversarial attacks was strongly correlated with increased vulnerability in low-selectivity models.
• The results demonstrate a fundamental trade-off: low selectivity enhances average-case robustness but compromises worst-case robustness due to high-dimensional representation shifts.