[Paper Review] Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems

Paul Smolensky, R. Thomas McCoy+3 more|arXiv (Cornell University)|May 2, 2022

Cognitive Science and Mapping4 citations

TL;DR

This paper introduces neurocompositional computing—a new AI paradigm that unifies the principles of compositionality and continuity to enable robust, interpretable, and generalizable intelligence. By leveraging Tensor Product Representations (TPRs) and a formalism called Neurally-Encoded Compositionally-Structured Tensor (NECST) computing, the approach achieves deeper compositional generalization and enhanced model interpretability, overcoming key limitations of standard deep learning systems.

ABSTRACT

What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome? The widely accepted narrative attributes this progress to massive increases in the quantity of computational and data resources available to support statistical learning in deep artificial neural networks. We show that an additional crucial factor is the development of a new type of computation. Neurocompositional computing adopts two principles that must be simultaneously respected to enable human-level cognition: the principles of Compositionality and Continuity. These have seemed irreconcilable until the recent mathematical discovery that compositionality can be realized not only through discrete methods of symbolic computing, but also through novel forms of continuous neural computing. The revolutionary recent progress in AI has resulted from the use of limited forms of neurocompositional computing. New, deeper forms of neurocompositional computing create AI systems that are more robust, accurate, and comprehensible.

Motivation & Objective

• To address the central paradox of cognition—how human-level intelligence emerges from continuous neural processes while preserving compositional structure—by identifying neurocompositional computing as the underlying computational framework.
• To diagnose the limitations of 20th-century and current 21st-century AI as stemming from architectures that fail to simultaneously respect both compositionality and continuity principles.
• To demonstrate that recent AI progress is partially due to the implicit emergence of neurocompositional computing, even if only in limited forms.
• To propose a deeper realization of neurocompositional computing through the NECST formalism, enabling more robust and interpretable AI systems.
• To show how NECST-based models improve generalization, controllability, and explainability by embedding compositional structure directly into continuous vector representations.

Proposed method

• Employing Tensor Product Representations (TPRs) to encode compositional structures in continuous vector spaces, allowing discrete symbolic-like structure to be embedded in continuous neural activations.
• Using binding and permutation operations in TPRs to represent complex, hierarchical structures such as phrases and sentences, enabling compositional generalization across novel combinations.
• Introducing the Neurally-Encoded Compositionally-Structured Tensor (NECST) formalism to systematically represent and process structured knowledge in continuous vector spaces with explicit compositional structure.
• Designing neural network architectures that are explicitly biased to exploit the disentangled representations of TPRs, promoting compositional generalization during training.
• Implementing optimization-based processing where symbol meanings adapt dynamically to structural context, enhancing inference and contextual sensitivity.
• Applying NECST to natural language processing tasks, enabling precise, interpretable modifications to internal representations that directly control output behavior through vector arithmetic.

Experimental results

Research questions

• RQ1 How can continuous neural computation simultaneously support compositional structure and continuous processing, resolving the long-standing tension between symbolic and connectionist approaches?
• RQ2 To what extent do current deep learning systems implicitly implement neurocompositional computing, and how does this explain their success despite known limitations?
• RQ3 Can a formalism like NECST enable deeper compositional generalization and improved interpretability in AI systems compared to standard deep learning?
• RQ4 How can neural networks be structured to actively exploit compositional structure in their representations, rather than learning it incidentally?
• RQ5 What role does context-sensitive, optimization-driven meaning adaptation play in enhancing the robustness and accuracy of AI systems?

Key findings

• Neurocompositional computing, realized through TPRs and NECST, enables AI systems to perform precise, interpretable modifications to internal representations using vector arithmetic, such as subtracting and adding meaning vectors to alter behavior.
• The NECST formalism allows for the embedding of nested and hierarchical structures (e.g., ‘unlockedable’) within continuous vector representations, enabling richer compositional generalization.
• By explicitly disentangling content and role in TPRs, models achieve better generalization and control, as demonstrated by accurate transformation of complex inputs like 8-word commands into 30-word outputs through successive vector operations.
• The approach enables high-accuracy behavioral control through internal modifications: for example, changing ‘jump twice’ to ‘jump thrice’ by modifying the vector encoding of ‘twice’ to that of ‘thrice’.
• NECST-based models show improved robustness and interpretability, reducing the opacity of standard deep learning systems and enhancing controllability in real-world applications.
• Future 3G neurocompositional computing systems will incorporate optimization-based processing, structural nesting, and architectural biases to fully exploit compositional structure, paving the way for major advances in language and reasoning.