[Paper Review] Evaluating Semantic Rationality of a Sentence: A Sememe-Word-Matching Neural Network based on HowNet

Shu Liu, Jingjing Xu+2 more|arXiv (Cornell University)|Sep 11, 2018

Topic Modeling19 references5 citations

TL;DR

This paper proposes a Sememe-Word-Matching Neural Network (SWM-NN) that leverages HowNet’s sememe knowledge base to evaluate sentence semantic rationality by modeling fine-grained semantic compatibility between words. The model outperforms baselines by 5.4% in accuracy on a newly constructed large-scale dataset, demonstrating improved detection of semantic violations through sememe-level reasoning.

ABSTRACT

Automatic evaluation of semantic rationality is an important yet challenging task, and current automatic techniques cannot well identify whether a sentence is semantically rational. The methods based on the language model do not measure the sentence by rationality but by commonness. The methods based on the similarity with human written sentences will fail if human-written references are not available. In this paper, we propose a novel model called Sememe-Word-Matching Neural Network (SWM-NN) to tackle semantic rationality evaluation by taking advantage of sememe knowledge base HowNet. The advantage is that our model can utilize a proper combination of sememes to represent the fine-grained semantic meanings of a word within the specific contexts. We use the fine-grained semantic representation to help the model learn the semantic dependency among words. To evaluate the effectiveness of the proposed model, we build a large-scale rationality evaluation dataset. Experimental results on this dataset show that the proposed model outperforms the competitive baselines with a 5.4\% improvement in accuracy.

Motivation & Objective

• To address the challenge of automatically detecting whether a sentence is semantically rational, especially when human-written references are unavailable.
• To overcome limitations of language model-based methods that prioritize sentence commonness over rationality.
• To develop a method that leverages fine-grained semantic knowledge to detect selectional restriction violations between predicates and their arguments.
• To construct a large-scale, human-annotated dataset for sentence semantic rationality detection (SSRD).
• To integrate word-level and sememe-level representations for improved semantic compatibility modeling.

Proposed method

• The model uses a two-branch architecture: one for word-level representation and another for sememe-level representation derived from HowNet.
• For each word, the model selects contextually appropriate sememes using a matching mechanism between word context and sememe definitions.
• Semantic compatibility between words is evaluated by computing the alignment and consistency of their selected sememes.
• The final rationality score is produced by fusing word-level and sememe-level representations through a neural network classifier.
• The model is trained end-to-end on a newly constructed dataset of sentences labeled for semantic rationality.
• A soft attention mechanism is used to dynamically weight sememe relevance based on sentence context.

Experimental results

Research questions

• RQ1 Can a neural network model effectively detect semantic rationality in sentences without relying on human-written references?
• RQ2 To what extent can sememe-level knowledge from HowNet improve the detection of semantic incoherence in sentences?
• RQ3 How does the integration of fine-grained semantic units (sememes) compare to word-level or sentence-level representations in rationality detection?
• RQ4 What is the performance gap between human judgment and automated models in semantic rationality evaluation?
• RQ5 How robust is the proposed model across different types of semantic violations, such as selectional restriction errors?

Key findings

• The proposed SWM-NN model achieves a 5.4% absolute improvement in accuracy over competitive baselines on the new large-scale rationality evaluation dataset.
• The model performs significantly better than language model-based methods, which often confuse uncommon but rational sentences with irrational ones.
• Human annotators show higher accuracy than the model, especially on the most challenging subset (dataset3), indicating room for improvement in modeling complex semantic constraints.
• The model’s performance is sensitive to the quality of sememe selection, highlighting the importance of context-aware sememe matching.
• The ablation study confirms that both word-level and sememe-level representations contribute significantly to the final prediction, with sememe-level features being particularly effective for detecting selectional restriction violations.
• The model generalizes well across different types of semantic errors, including those involving incorrect semantic roles or incompatible semantic features.