[Paper Review] Context is Everything: Finding Meaning Statistically in Semantic Spaces.

Eric Zelikman|arXiv (Cornell University)|Mar 22, 2018

Topic Modeling14 references3 citations

TL;DR

This paper proposes Contextual Salience (CoSal), a statistically grounded measure of word importance in context that outperforms tf-idf and state-of-the-art models in sentence embedding tasks. By leveraging small contexts and phrase-level semantics, CoSal enables more accurate, context-aware representations for downstream NLP tasks like sentence similarity, summarization, and cosine distance, achieving superior performance on Facebook's SentEval benchmarks.

ABSTRACT

This paper introduces Contextual Salience (CoSal), a simple and explicit measure of a word's importance in context which is a more theoretically natural, practically simpler, and more accurate replacement to tf-idf. CoSal supports very small contexts (20 or more sentences), out-of context words, and is easy to calculate. A word vector space generated with both bigram phrases and unigram tokens reveals that contextually significant words disproportionately define phrases. This relationship is applied to produce simple weighted bag-of-words sentence embeddings. This model outperforms SkipThought and the best models trained on unordered sentences in most tests in Facebook's SentEval, beats tf-idf on all available tests, and is generally comparable to the state of the art. This paper also applies CoSal to sentence and document summarization and an improved and context-aware cosine distance. Applying the premise that unexpected words are important, CoSal is presented as a replacement for tf-idf and an intuitive measure of contextual word importance.

Motivation & Objective

• To address the limitations of tf-idf in capturing contextually relevant word importance.
• To develop a simple, theoretically sound, and computationally efficient alternative to tf-idf for measuring word significance in context.
• To improve sentence and document representation learning by incorporating contextually salient words.
• To enhance sentence embedding quality through weighted bag-of-words modeling using CoSal.
• To enable better performance in downstream NLP tasks such as sentence similarity, summarization, and cosine distance computation.

Proposed method

• Defining Contextual Salience (CoSal) as a statistical measure of word importance based on co-occurrence patterns within small contexts (20+ sentences).
• Constructing a word vector space using both unigram tokens and bigram phrases to capture phrase-level semantics.
• Applying CoSal to weight bag-of-words sentence embeddings, producing contextually informed sentence representations.
• Using CoSal to improve sentence and document summarization by prioritizing contextually significant words.
• Introducing a context-aware cosine distance metric that incorporates CoSal to better reflect semantic similarity.
• Validating CoSal's effectiveness through comparisons with tf-idf and state-of-the-art models on Facebook's SentEval benchmark.

Experimental results

Research questions

• RQ1 Can a contextually grounded measure of word importance outperform tf-idf in capturing semantic relevance?
• RQ2 How does CoSal improve sentence embedding quality compared to existing methods like SkipThought and unordered sentence models?
• RQ3 To what extent do contextually salient words define and distinguish phrases in semantic spaces?
• RQ4 Can CoSal enhance performance in sentence and document summarization tasks?
• RQ5 Does a context-aware cosine distance based on CoSal improve semantic similarity estimation?

Key findings

• CoSal outperforms tf-idf on all available tests in Facebook's SentEval benchmark, demonstrating consistent superiority in word importance estimation.
• The CoSal-based sentence embedding model achieves performance comparable to the state of the art across most SentEval tasks.
• Contextually significant words are disproportionately responsible for defining phrases in the learned vector space.
• The model improves sentence and document summarization by emphasizing semantically salient terms based on context.
• The context-aware cosine distance, informed by CoSal, provides a more accurate measure of semantic similarity than standard cosine distance.
• CoSal enables effective representation learning even with very small contexts (20+ sentences), making it practical for real-world applications.