[Paper Review] RNNs Implicitly Implement Tensor Product Representations

R. Thomas McCoy, Tal Linzen+2 more|arXiv (Cornell University)|Dec 20, 2018

Topic Modeling41 citations

TL;DR

The paper introduces Tensor Product Decomposition Networks (TPDNs) to show that RNN encodings of sequences and sentences can be approximated by tensor product representations, revealing when and how structure emerges in neural representations.

ABSTRACT

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated by TPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations.

Motivation & Objective

• Motivate the idea that RNNs may learn compositional symbolic structure.
• Propose and validate the Tensor Product Decomposition Network (TPDN) to approximate existing vector encodings with tensor product representations.

Proposed method

• Define and implement TPDN to learn fillers and roles that reconstruct given encodings via tensor products.
• Test TPDN on synthetic digit sequence autoencoders across unidirectional, bidirectional, and tree-based architectures.
• Evaluate whether natural language sentence encoders' representations are structure-sensitive or largely bag-of-words-like.
• Use substitution accuracy to measure how well a TPDN approximates a given encoder without further training the decoder.
• Analyze how architecture and training task influence the emergence of structure in representations.

Experimental results

Research questions

• RQ1 Can RNN encodings of sequences be well approximated by tensor product representations under appropriate role schemes?
• RQ2 Do different RNN architectures (unidirectional, bidirectional, tree-based) induce distinct structural representations?
• RQ3 Are sentence-embedding models’ representations largely structure-free (bag-of-words) or do they exhibit robust structure detectable by TPRs?
• RQ4 How do encoder vs. decoder choices, and training tasks, affect the learnability of compositional representations?
• RQ5 What training conditions foster or hinder the emergence of structure-sensitive representations in RNNs?

Key findings

• TPDNs can closely approximate encodings from certain RNN architectures with structure-sensitive role schemes (e.g., tree-position roles for tree-based autoencoders, bidirectional roles for unidirectional ones).
• Unidirectional and tree-based autoencoders are well approximated by corresponding role schemes, while bidirectional encoders are not consistently well captured by the tested schemes.
• For natural language sentence encoders (InferSent, Skip-thought, SST, SPINN), bag-of-words roles provide a strong, sometimes near-equivalent approximation, suggesting limited robust structure.
• The decoder architecture tends to have a larger influence on the learned representations than the encoder, with tree-based decoders yielding tree-based role structure.
• Training tasks that require explicit structure (e.g., nontrivial ordering or interleaving) promote stronger structure-sensitive representations, whereas tasks like sorting tend toward bag-of-words representations.