[Paper Review] Reference-less Measure of Faithfulness for Grammatical Error Correction

Leshem Choshen, Omri Abend|arXiv (Cornell University)|Apr 11, 2018

Natural Language Processing Techniques24 references4 citations

TL;DR

This paper proposes USim, a reference-less semantic faithfulness measure for Grammatical Error Correction (GEC) that evaluates how well a correction preserves the meaning of the original sentence by comparing their UCCA-structured semantic representations. The method achieves high similarity scores for valid corrections and significantly lower scores for semantically unfaithful ones, demonstrating its effectiveness in detecting meaning-preserving edits without requiring human-annotated references.

ABSTRACT

We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.

Motivation & Objective

• To address the limitations of reference-based evaluation in GEC, which are costly and biased due to the vast space of valid corrections.
• To develop a reliable, reference-less measure that evaluates semantic faithfulness—how well a correction preserves the original meaning.
• To validate that semantic annotation can be consistently applied to learner language, even when syntactically ungrammatical.
• To demonstrate that USim can distinguish between semantically faithful and unfaithful corrections without relying on human-annotated references.
• To complement existing reference-less measures focused on grammaticality by introducing a measure focused on meaning preservation.

Proposed method

• USim computes a graph distance between the UCCA semantic structures of the source sentence and the corrected output.
• UCCA structures are represented as directed acyclic graphs (DAGs), where nodes are semantic units and edges denote semantic roles.
• The method uses a pre-trained UCCA parser (TUPA) to automatically generate semantic representations from raw text.
• Semantic faithfulness is measured by comparing the structural similarity of the source and corrected sentence’s UCCA graphs.
• The approach is adaptable to other semantic schemes like AMR, though UCCA is used due to its broad predicate coverage and robustness across linguistic variations.
• The method avoids reliance on word-level alignment or lexical overlap, focusing instead on structural and semantic coherence.

Experimental results

Research questions

• RQ1 Can UCCA semantic structures be consistently and automatically applied to ungrammatical learner language text?
• RQ2 Does USim assign higher similarity scores to valid corrections than to invalid or semantically divergent ones?
• RQ3 Is USim sensitive to semantic unfaithfulness, even when grammaticality is preserved?
• RQ4 Can USim reliably detect meaning-preserving edits without requiring human-annotated references?
• RQ5 How does USim compare to reference-based metrics in capturing semantic faithfulness in GEC outputs?

Key findings

• UCCA parsing technology is mature enough to enable reliable automatic semantic representation of learner language, with consistent application across ungrammatical sentences.
• Valid corrections consistently receive high USim scores (e.g., 0.71–0.78 for references), indicating strong semantic preservation.
• Invalid or semantically divergent corrections receive significantly lower USim scores, with values as low as 0.19–0.39 for partially trained GEC systems.
• Even when a correction makes more lexical changes than the reference, it can still receive a lower USim score if it alters the semantic structure, demonstrating sensitivity to meaning.
• USim successfully identifies that a correction like 'get on' instead of 'get the idea' is semantically less faithful, assigning it a score of 0.33 versus 0.71 for the reference.
• The method shows that semantic faithfulness is not compromised by grammatical improvements alone, and that USim effectively penalizes changes that distort meaning.