[Paper Review] Clustering units in neural networks: upstream vs downstream information

Richard D. Lange, David Rolnick+1 more|arXiv (Cornell University)|Mar 22, 2022

Machine Learning in Materials Science4 citations

TL;DR

This paper proposes a framework to detect functional modules in neural networks by clustering hidden units based on upstream (input-driven) or downstream (output-influencing) functional similarity. It finds that dropout significantly increases modularity, and that upstream and downstream similarity measures yield largely non-overlapping clusters, revealing a fundamental misalignment between input-driven and output-driven interpretations of neural representations.

ABSTRACT

It has been hypothesized that some form of "modular" structure in artificial neural networks should be useful for learning, compositionality, and generalization. However, defining and quantifying modularity remains an open problem. We cast the problem of detecting functional modules into the problem of detecting clusters of similar-functioning units. This begs the question of what makes two units functionally similar. For this, we consider two broad families of methods: those that define similarity based on how units respond to structured variations in inputs ("upstream"), and those based on how variations in hidden unit activations affect outputs ("downstream"). We conduct an empirical study quantifying modularity of hidden layer representations of simple feedforward, fully connected networks, across a range of hyperparameters. For each model, we quantify pairwise associations between hidden units in each layer using a variety of both upstream and downstream measures, then cluster them by maximizing their "modularity score" using established tools from network science. We find two surprising results: first, dropout dramatically increased modularity, while other forms of weight regularization had more modest effects. Second, although we observe that there is usually good agreement about clusters within both upstream methods and downstream methods, there is little agreement about the cluster assignments across these two families of methods. This has important implications for representation-learning, as it suggests that finding modular representations that reflect structure in inputs (e.g. disentanglement) may be a distinct goal from learning modular representations that reflect structure in outputs (e.g. compositionality).

Motivation & Objective

• To operationalize functional modules in neural networks as clusters of similarly behaving hidden units.
• To investigate how different definitions of functional similarity—upstream (input-driven) vs. downstream (output-influencing)—affect module detection.
• To evaluate the impact of regularization techniques like dropout, weight decay, and L1 penalties on modularity.
• To assess the alignment between upstream and downstream definitions of neural unit similarity.
• To challenge assumptions about neural representations being uniformly interpretable via input or output perspectives.

Proposed method

• Quantify pairwise functional similarity between hidden units using eight distinct upstream and downstream measures.
• Construct an n×n similarity matrix for each layer, where entries reflect functional similarity between unit pairs.
• Apply network science-based clustering algorithms to maximize modularity score Q*, identifying blocks of high intra-cluster similarity.
• Use established modularity optimization tools (e.g., Louvain algorithm) to assign units to clusters based on similarity matrices.
• Train over 250 fully connected feedforward networks on MNIST to evaluate robustness across hyperparameters.
• Compare cluster assignments across upstream and downstream methods to assess consistency and alignment.

Experimental results

Research questions

• RQ1 How do different measures of functional similarity—upstream versus downstream—compare in identifying clusters of similarly functioning units in neural networks?
• RQ2 To what extent does dropout increase modularity compared to other regularization techniques like weight decay or L1 penalties?
• RQ3 Are the clusters identified via upstream similarity measures aligned with those from downstream similarity measures?
• RQ4 Does the functional modularity of a network correlate with downstream performance or generalization?
• RQ5 Can upstream notions of disentanglement (e.g., feature-specific unit activation) be meaningfully aligned with downstream causal roles in prediction?

Key findings

• Dropout significantly increases modularity (as measured by Q*) compared to other regularization methods, suggesting it induces redundancy that enhances cluster formation.
• The main effect of dropout on modularity arises from creating near-identical copies of units, which behave similarly under dropout, leading to higher functional similarity.
• Upstream and downstream similarity measures yield largely non-overlapping cluster assignments, indicating a fundamental disconnect between input-driven and output-driven interpretations of neural function.
• Despite good internal consistency within upstream and within downstream methods, there is minimal agreement across the two families of similarity measures.
• The results suggest that disentangled representations (upstream) and causally effective representations (downstream) are distinct concepts, even in trained networks.
• Maximizing modularity during training is challenging due to local minima and feedback between cluster assignments and neural activity, suggesting soft clustering may be necessary.