[Paper Review] Integrating Multiplicative Features into Supervised Distributional Methods for Lexical Entailment

Tu Vu, Vered Shwartz|arXiv (Cornell University)|Apr 24, 2018

Natural Language Processing Techniques18 references3 citations

TL;DR

This paper proposes integrating multiplicative features—specifically element-wise multiplication (Mult) and squared difference (Sqdiff)—into standard supervised distributional representations for lexical entailment. By combining these features with existing representations like Concat and Diff, the method significantly boosts performance, especially in strict evaluation settings, with Concat⊕Mult emerging as a strong, consistent baseline for linear classifiers.

ABSTRACT

Supervised distributional methods are applied successfully in lexical entailment, but recent work questioned whether these methods actually learn a relation between two words. Specifically, Levy et al. (2015) claimed that linear classifiers learn only separate properties of each word. We suggest a cheap and easy way to boost the performance of these methods by integrating multiplicative features into commonly used representations. We provide an extensive evaluation with different classifiers and evaluation setups, and suggest a suitable evaluation setup for the task, eliminating biases existing in previous ones.

Motivation & Objective

• Address concerns that supervised distributional methods for lexical entailment may rely on lexical memorization rather than learning relational semantics.
• Investigate whether multiplicative features can enhance the ability of linear and non-linear classifiers to capture relational information between word pairs.
• Evaluate the effectiveness of various representations and classifiers under multiple evaluation setups to identify a reliable, unbiased benchmark for lexical entailment.
• Propose a suitable evaluation setup that minimizes data leakage and memorization, particularly focusing on out-of-domain and lexical-split evaluations.
• Establish Concat⊕Mult as a strong, general-purpose baseline for future research in supervised lexical entailment.

Proposed method

• Introduce two multiplicative features: Mult = v_x ⊙ v_y (element-wise multiplication) and Sqdiff = (v_y - v_x) ⊙ (v_y - v_x) (squared difference).
• Integrate these features into standard word-pair representations, such as Concat(v_x ⊕ v_y), Diff(v_y - v_x), Sum(v_x + v_y), and their combinations.
• Use both linear (SVM with linear kernel) and non-linear (SVM with RBF kernel) classifiers to evaluate performance across different setups.
• Apply the representations in multiple evaluation scenarios: standard, lexical-split (LEX), and out-of-domain (OOD), to assess robustness and generalization.
• Analyze feature contributions through qualitative examples and cosine similarity analysis to understand when and why multiplicative features improve classification.
• Propose and advocate for OOD evaluation as the most suitable benchmark, as it reduces memorization bias and better reflects true relational learning.

Experimental results

Research questions

• RQ1 Can multiplicative features like Mult and Sqdiff improve the performance of supervised distributional methods in lexical entailment?
• RQ2 Does integrating multiplicative features help mitigate the issue of lexical memorization, especially in strict evaluation settings like LEX and OOD?
• RQ3 How do different representations (e.g., Concat, Diff, Sum) combined with multiplicative features compare in terms of performance and robustness?
• RQ4 Is the performance gain from multiplicative features more pronounced in linear or non-linear classifiers, and under which evaluation conditions?
• RQ5 What is the most appropriate evaluation setup for lexical entailment that minimizes data leakage and ensures genuine relational learning?

Key findings

• Integrating multiplicative features, especially Mult, significantly boosts the performance of linear classifiers in lexical entailment, with gains most prominent in strict evaluation setups.
• Concat⊕Mult consistently outperforms other combinations across multiple datasets and classifiers, establishing it as a strong, reliable baseline.
• In out-of-domain (OOD) and lexical-split (LEX) evaluations—where memorization is disabled—multiplicative features yield the most substantial performance improvements.
• The addition of Mult helps distinguish between related and unrelated word pairs by capturing semantic similarity, as evidenced by cosine similarity analysis showing clear separation between random (0.141) and other relations (0.279–0.426).
• Pairs with low similarity scores that were misclassified by Concat alone (e.g., classifying unrelated adjectives as attribute) are correctly labeled as random when Mult is added.
• Even non-linear classifiers benefit more from multiplicative features in strict setups, though the gains are more substantial for linear models, which remain efficient and effective when enhanced this way.