[Paper Review] Generalizing Outside the Training Set: When Can Neural Networks Learn Identity Effects?

Simone Brugiapaglia, Matthew Tingchi Liu+1 more|arXiv (Cornell University)|May 9, 2020

Neural Networks and Applications6 references4 citations

TL;DR

This paper investigates whether neural networks can generalize identity effects—such as recognizing that 'AA' is grammatical while 'AB' is not—beyond their training data. Using a theoretical framework based on input encoding invariance, it shows that standard one-hot and orthogonal encodings prevent generalization, while distributed encodings like 3-active-bit allow partial generalization, demonstrating that encoding choice critically determines whether networks can learn abstract algebraic rules without explicit variable instantiation.

ABSTRACT

Often in language and other areas of cognition, whether two components of an object are identical or not determine whether it is well formed. We call such constraints identity effects. When developing a system to learn well-formedness from examples, it is easy enough to build in an identify effect. But can identity effects be learned from the data without explicit guidance? We provide a simple framework in which we can rigorously prove that algorithms satisfying simple criteria cannot make the correct inference. We then show that a broad class of algorithms including deep neural networks with standard architecture and training with backpropagation satisfy our criteria, dependent on the encoding of inputs. Finally, we demonstrate our theory with computational experiments in which we explore the effect of different input encodings on the ability of algorithms to generalize to novel inputs.

Motivation & Objective

• To determine whether neural networks can learn identity effects—such as distinguishing identical from different elements—when tested on novel inputs not seen during training.
• To investigate how input encoding influences the ability of learning algorithms to generalize beyond training data.
• To formalize conditions under which learning algorithms are invariant to transformations that preserve identity effects.
• To bridge theoretical analysis with computational experiments to validate predictions about generalization performance across different encodings.

Proposed method

• The authors define a learning algorithm as a function L(D, w) that maps a training dataset D and input w to a rating, combining model training and inference into a single function.
• They introduce a transformation τ on inputs that preserves identity effects (e.g., swapping letters not in the training set), and define invariance of the algorithm and dataset under τ.
• A key theoretical result establishes that if both the training data D and the learning algorithm L are invariant under τ, then L(D, w₁) = L(D, w₂) for any w₁, w₂ related by τ, meaning the model cannot distinguish them.
• The paper proves that standard deep neural networks with backpropagation training are invariant under τ when inputs are encoded via one-hot or orthogonal methods like Haar encoding.
• Computational experiments compare three encodings—one-hot, 3-active-bit distributed, and Haar—on a binary classification task for identity effects.
• The experiments evaluate generalization by measuring average ratings for novel inputs like 'YY' (should be 1) and 'YZ' (should be 0), assessing whether models assign higher scores to identity pairs.

Experimental results

Research questions

• RQ1 Can neural networks learn identity effects—such as distinguishing identical from non-identical pairs—on novel inputs not present in the training set?
• RQ2 Under what conditions does input encoding prevent or enable generalization of identity effects in neural networks?
• RQ3 Does the invariance of the learning algorithm and training data under a transformation τ imply that the model cannot distinguish between inputs related by τ?
• RQ4 To what extent can distributed encodings like 3-active-bit allow partial generalization when one-hot or orthogonal encodings fail?
• RQ5 Is there a measurable difference in model performance on novel identity pairs versus non-identity pairs when using different input encodings?

Key findings

• Neural networks trained with one-hot encoding failed to generalize, assigning similar ratings to both 'YY' and 'YZ', indicating no distinction between identity and non-identity pairs.
• With Haar encoding, which is orthogonal, networks also failed to generalize, showing no meaningful difference in ratings between 'YY' and 'YZ'.
• In contrast, the 3-active-bit distributed encoding enabled partial generalization, with networks assigning significantly higher average ratings to 'YY' than to 'YZ'.
• The training and validation loss curves showed a clear gap in the one-hot and Haar cases, indicating poor generalization, while the distributed encoding showed no such gap.
• Theoretical analysis confirmed that orthogonal encodings lead to invariance under τ, explaining the failure to generalize, while the non-orthogonal 3-active-bit encoding breaks this invariance, allowing partial generalization.
• The results support the claim that encoding structure determines whether neural networks can generalize algebraic rules like identity effects without explicit variable instantiation.