[Paper Review] Learning Semantically and Additively Compositional Distributional Representations

Ran Tian, Naoaki Okazaki+1 more|arXiv (Cornell University)|Jun 8, 2016

Natural Language Processing Techniques52 references3 citations

TL;DR

This paper proposes a vector-based compositional model that links distributional semantics with Dependency-based Compositional Semantics (DCS), realizing logical operations like intersection and projection through vector addition and linear transformations. It achieves near state-of-the-art performance on phrase similarity and relation classification tasks, and sets a new state-of-the-art on sentence completion by enabling executable, semantically compositional vector queries.

ABSTRACT

This paper connects a vector-based composition model to a formal semantics, the Dependency-based Compositional Semantics (DCS). We show theoretical evidence that the vector compositions in our model conform to the logic of DCS. Experimentally, we show that vector-based composition brings a strong ability to calculate similar phrases as similar vectors, achieving near state-of-the-art on a wide range of phrase similarity tasks and relation classification; meanwhile, DCS can guide building vectors for structured queries that can be directly executed. We evaluate this utility on sentence completion task and report a new state-of-the-art.

Motivation & Objective

• To bridge distributional vector semantics with formal compositional semantics, specifically Dependency-based Compositional Semantics (DCS), to enable logical composition in vector space.
• To develop a joint training framework that learns word vectors and projection matrices such that vector compositions mirror DCS operations like intersection and projection.
• To enable vector-based queries that can be directly executed, such as retrieving candidate answers via dot product similarity.
• To improve phrase similarity and relation classification by leveraging syntactic-semantic roles through role-specific linear transformations.
• To demonstrate the utility of compositional vector representations in natural language inference and sentence completion tasks.

Proposed method

• Replace DCS denotations with word vectors and realize logical operations using vector arithmetic: intersection via addition and projection via learned linear transformations (matrices).
• Train word vectors and projection matrices jointly from unlabeled corpora using a differentiable objective that enforces compositional consistency with DCS logic.
• Use matrix multiplication to project vectors based on syntactic-semantic roles (e.g., COMP, SUBJ), with each role associated with a distinct matrix (e.g., M_COMP).
• Construct phrase vectors by adding the projected verb vector to the head noun vector, emulating the DCS tree computation: e.g., v_banned_drugs = M_COMP × v_ban + v_drug.
• Enable query execution by using the resulting phrase vector to compute dot products with candidate answer vectors, ranking them by similarity.
• Leverage additive compositionality as a learning signal, grounded in the observation that overlapping contexts approximate vector addition.

Experimental results

Research questions

• RQ1 Can vector-based composition be formally linked to the logical operations of Dependency-based Compositional Semantics (DCS) such as intersection and projection?
• RQ2 Does additive composition of word vectors approximate the logical intersection operation in DCS, and can this be learned jointly with syntactic role projections?
• RQ3 Can the resulting vector representations achieve state-of-the-art performance on phrase similarity and relation classification tasks?
• RQ4 Can the learned vector representations support executable queries, such as in sentence completion, by enabling direct vector-based retrieval?
• RQ5 Can the model learn role-specific matrices that distinguish syntactic-semantic roles (e.g., COMP vs. SUBJ) while preserving compositional meaning?

Key findings

• The model achieves near state-of-the-art performance on multiple phrase similarity benchmarks, demonstrating strong compositional generalization.
• The model sets a new state-of-the-art on the sentence completion task by using learned vector queries to retrieve semantically appropriate fillers.
• The joint training of word vectors and projection matrices yields representations that preserve syntactic-semantic roles, with distinct matrices for different grammatical functions.
• The model’s vector compositions conform to the logical structure of DCS, providing theoretical grounding for compositional vector semantics.
• The use of additive composition for intersection and linear mapping for projection enables both semantic similarity and executable query capabilities.
• The approach outperforms GloVe and dependency-based vector models in capturing compositional meaning and role-specific semantics.