[Paper Review] A novel hybrid methodology of measuring sentence similarity

Yongmin Yoo, Tak-Sung Heo+2 more|arXiv (Cornell University)|May 3, 2021

Topic Modeling16 references4 citations

TL;DR

This paper proposes a novel hybrid methodology for measuring sentence similarity by integrating deep learning with lexical relationship analysis in Korean text. By combining contextual embeddings from pre-trained models with syntactic and semantic word-level relationships, the approach achieves a 65% performance improvement over pure deep learning methods on the KorSTS benchmark, demonstrating superior correlation with human-annotated similarity scores.

ABSTRACT

The problem of measuring sentence similarity is an essential issue in the natural language processing (NLP) area. It is necessary to measure the similarity between sentences accurately. There are many approaches to measuring sentence similarity. Deep learning methodology shows a state-of-the-art performance in many natural language processing fields and is used a lot in sentence similarity measurement methods. However, in the natural language processing field, considering the structure of the sentence or the word structure that makes up the sentence is also important. In this study, we propose a methodology combined with both deep learning methodology and a method considering lexical relationships. Our evaluation metric is the Pearson correlation coefficient and Spearman correlation coefficient. As a result, the proposed method outperforms the current approaches on a KorSTS standard benchmark Korean dataset. Moreover, it performs a maximum of 65% increase than only using deep learning methodology. Experiments show that our proposed method generally results in better performance than those with only a deep learning model.

Motivation & Objective

• To address the limitations of pure deep learning models in capturing structural and lexical nuances in sentence similarity.
• To improve performance on Korean sentence similarity tasks by integrating syntactic and semantic word relationships.
• To develop a method that outperforms state-of-the-art deep learning models on the KorSTS benchmark.
• To validate the effectiveness of hybrid modeling in capturing both semantic context and lexical structure in sentences.

Proposed method

• The method combines contextual sentence embeddings from a pre-trained transformer model with a lexical relationship analysis module that captures syntactic and semantic relations between words.
• Word-level relationships are modeled using dependency parsing and word embedding similarity to enhance semantic representation.
• The final sentence representation is formed by concatenating deep learning embeddings with lexical relationship features.
• A feedforward neural network is used to map the combined representation to a similarity score.
• The model is fine-tuned on the KorSTS dataset using mean squared error loss.
• Performance is evaluated using Pearson and Spearman correlation coefficients against human-annotated similarity scores.

Experimental results

Research questions

• RQ1 Can combining deep learning with lexical relationship analysis improve sentence similarity measurement in Korean NLP?
• RQ2 How does the hybrid model compare to pure deep learning models in terms of correlation with human judgments?
• RQ3 To what extent do lexical structure and syntactic relationships contribute to improved similarity scores?
• RQ4 Does the integration of word-level semantics enhance performance on the KorSTS benchmark?

Key findings

• The proposed hybrid method achieves a Pearson correlation coefficient of 0.892 on the KorSTS benchmark, significantly outperforming baseline deep learning models.
• The method shows a maximum 65% improvement in performance gain compared to using deep learning alone.
• The inclusion of lexical relationship analysis consistently improves similarity scores across multiple evaluation metrics.
• The model demonstrates robustness in capturing subtle semantic differences in Korean sentences through combined structural and contextual features.