[Paper Review] Dis-S2V: Discourse Informed Sen2Vec

Tanay Kumar Saha, Shafiq Joty+2 more|arXiv (Cornell University)|Oct 25, 2016

Topic Modeling25 references3 citations

TL;DR

This paper proposes Dis-S2V, a novel method for learning sentence representations by integrating discourse-level relations into content-based Sen2Vec models. It uses a discourse graph to encode inter-sentence relations and applies retrofitting or graph regularization to enhance vector representations, significantly improving performance on classification, clustering, and ranking tasks compared to content-only baselines.

ABSTRACT

Vector representation of sentences is important for many text processing tasks that involve clustering, classifying, or ranking sentences. Recently, distributed representation of sentences learned by neural models from unlabeled data has been shown to outperform the traditional bag-of-words representation. However, most of these learning methods consider only the content of a sentence and disregard the relations among sentences in a discourse by and large. In this paper, we propose a series of novel models for learning latent representations of sentences (Sen2Vec) that consider the content of a sentence as well as inter-sentence relations. We first represent the inter-sentence relations with a language network and then use the network to induce contextual information into the content-based Sen2Vec models. Two different approaches are introduced to exploit the information in the network. Our first approach retrofits (already trained) Sen2Vec vectors with respect to the network in two different ways: (1) using the adjacency relations of a node, and (2) using a stochastic sampling method which is more flexible in sampling neighbors of a node. The second approach uses a regularizer to encode the information in the network into the existing Sen2Vec model. Experimental results show that our proposed models outperform existing methods in three fundamental information system tasks demonstrating the effectiveness of our approach. The models leverage the computational power of multi-core CPUs to achieve fine-grained computational efficiency. We make our code publicly available upon acceptance.

Motivation & Objective

• To improve sentence representation learning by incorporating discourse-level inter-sentence relations, which are often ignored in existing content-only models.
• To address the challenge of effectively modeling and leveraging discourse structure in sentence representation learning.
• To evaluate the impact of discourse information on downstream NLP tasks such as classification, clustering, and ranking.
• To develop efficient, computationally lightweight models that scale well on multi-core CPUs.

Proposed method

• Construct a discourse graph where nodes represent sentences and edges represent sentence similarity, capturing inter-sentence relations.
• Use the node2vec algorithm to pre-train sentence representations on the discourse graph, capturing structural and relational information.
• Apply two retrofitting strategies: (i) iterative adjacency-based retrofitting using 1-hop neighbors, and (ii) a discriminative sampling method for more flexible neighbor selection.
• Introduce a graph-smoothing regularizer into the original Sen2Vec objective function to jointly optimize content and discourse information.
• Combine the content-based Sen2Vec model (using Skip-gram or DBOW) with discourse-aware representations through retrofitting or regularization.
• Leverage multi-core CPU parallelism to achieve fine-grained computational efficiency in training and inference.

Experimental results

Research questions

• RQ1 Can discourse relations between sentences improve the quality of distributed sentence representations?
• RQ2 How can inter-sentence relations be effectively modeled and integrated into existing content-based sentence embedding frameworks?
• RQ3 Does incorporating discourse structure lead to measurable improvements in downstream NLP tasks such as classification, clustering, and ranking?
• RQ4 Which method—retrofitting or regularization—better preserves semantic content while enhancing discourse-awareness?

Key findings

• The proposed Dis-S2V models outperform the content-only Sen2Vec baseline across all three tasks: classification, clustering, and ranking.
• The regularized model (with graph-smoothing regularizer) achieved strong performance across all tasks, indicating robust integration of discourse information.
• The retrofitting-based models showed significant gains in classification and clustering, but were less effective in ranking tasks.
• The discourse graph representation effectively captured both topical and logical relations between sentences, enhancing semantic coherence in sentence vectors.
• The models achieved high computational efficiency and scalability on multi-core CPUs, making them practical for large-scale applications.