[Paper Review] Discriminative Phrase Embedding for Paraphrase Identification

Wenpeng Yin, Hinrich Schütze|arXiv (Cornell University)|Apr 2, 2016

Natural Language Processing Techniques3 citations

TL;DR

This paper proposes TF-KLD-KNN, a discriminative weighting scheme that extends TF-KLD to handle out-of-vocabulary units via k-nearest neighbors in embedding space, and introduces joint learning of continuous and discontinuous phrase embeddings. By combining these with weighted sentence vector sums using cosine similarity, element-wise sum, and difference features, the method achieves state-of-the-art performance on the MSRP paraphrase identification benchmark, outperforming prior systems by up to 0.052 in F1 on the subset.

ABSTRACT

This work, concerning paraphrase identification task, on one hand contributes to expanding deep learning embeddings to include continuous and discontinuous linguistic phrases. On the other hand, it comes up with a new scheme TF-KLD-KNN to learn the discriminative weights of words and phrases specific to paraphrase task, so that a weighted sum of embeddings can represent sentences more effectively. Based on these two innovations we get competitive state-of-the-art performance on paraphrase identification.

Motivation & Objective

• To improve sentence representation for paraphrase identification by incorporating continuous and discontinuous phrases beyond single words.
• To address data sparsity in feature weighting by extending TF-KLD to unknown units using k-nearest neighbors in embedding space.
• To achieve state-of-the-art performance on the MSRP paraphrase identification dataset through enhanced sentence vector composition and feature engineering.
• To evaluate the effectiveness of linguistic versus statistical phrase detection methods in the context of paraphrase identification.

Proposed method

• Learns distributed embeddings for single words, continuous phrases (e.g., 'side effects'), and discontinuous phrases (e.g., 'pick ... off') using a large lexicon-based phrase collection from Wiktionary and WordNet.
• Applies a heuristic based on co-occurrence statistics to classify phrases as continuous or discontinuous.
• Proposes TF-KLD-KNN: a reweighting scheme that computes weights for unknown units as the average of the k-nearest neighbors' weights in the embedding space using cosine similarity.
• Represents sentences as weighted sums of unit embeddings, where weights are derived from TF-KLD-KNN, and computes three types of sentence pair features: cosine similarity, element-wise sum, and absolute difference.
• Combines these learned features with standard machine translation metrics (MT features) and trains a linear SVM for classification.
• Uses word2vec skip-gram with word2phrase to generate statistical phrase embeddings for comparison with linguistically defined phrases.

Experimental results

Research questions

• RQ1 Can the inclusion of continuous and discontinuous phrases improve sentence representation for paraphrase identification?
• RQ2 How effectively can TF-KLD-KNN handle out-of-vocabulary units in the context of discriminative feature weighting?
• RQ3 Does using linguistically motivated phrases outperform statistically derived phrases in paraphrase identification?
• RQ4 To what extent do combined features (cosine, sum, difference) improve performance over individual features?

Key findings

• WORD+PHRASE, which uses both word and phrase embeddings with TF-KLD-KNN weighting, achieves the highest F1 score of 0.848 on the overall MSRP dataset, significantly outperforming the MT baseline (p < 0.05).
• On the smaller subset, WORD+PHRASE achieves an F1 of 0.857, outperforming the MT baseline by 0.028 and the next-best system by 0.008.
• TF-KLD-KNN achieves the highest performance (F1 = 0.848) among all reweighting schemes, outperforming TF-IDF (F1 = 0.821) and TF-KLD (F1 = 0.842).
• The KNN-based reweighting scheme for unseen units outperforms zero, type-average, and context-average baselines, confirming its effectiveness in handling data sparsity.
• WORD+GOOGLE (using statistical phrases from word2vec) performs slightly worse than WORD+PHRASE, suggesting linguistic phrases are more effective than statistical ones for this task.
• The performance gap between overall and subset results for WORD+PHRASE is explained by the lack of phrase content in sentences without phrases, indicating that phrase embeddings only help when phrases are present.