[Paper Review] Does Correction Remain A Problem For Large Language Models?

Xiaowu Zhang, Xiaotian Zhang+3 more|arXiv (Cornell University)|Aug 3, 2023

Topic Modeling4 citations

TL;DR

This paper investigates whether large language models (LLMs) like GPT-3.5-Turbo can effectively perform text correction and tolerate noisy inputs. Using few-shot prompting on Chinese spelling correction (CSC), Chinese text correction (CTC), and English grammar correction (GEC) datasets—including a newly curated 1,000-sample CSC dataset—it finds that LLMs achieve high accuracy in correcting errors while optimizing sentence expression, though they occasionally over-correct by altering meaning or structure. LLMs show strong tolerance to spelling errors (up to 5%) but degrade significantly when structural changes like word insertion or deletion are introduced.

ABSTRACT

As large language models, such as GPT, continue to advance the capabilities of natural language processing (NLP), the question arises: does the problem of correction still persist? This paper investigates the role of correction in the context of large language models by conducting two experiments. The first experiment focuses on correction as a standalone task, employing few-shot learning techniques with GPT-like models for error correction. The second experiment explores the notion of correction as a preparatory task for other NLP tasks, examining whether large language models can tolerate and perform adequately on texts containing certain levels of noise or errors. By addressing these experiments, we aim to shed light on the significance of correction in the era of large language models and its implications for various NLP applications.

Motivation & Objective

• To evaluate whether large language models (LLMs) can serve as effective text correction models for tasks like Chinese spelling correction (CSC), Chinese text correction (CTC), and English grammar correction (GEC).
• To assess the robustness of LLMs when processing datasets containing varying levels of noise, including spelling errors and structural alterations.
• To investigate whether LLMs over-correct or under-correct by modifying sentence structure or meaning during correction tasks.
• To improve the reliability of text correction evaluation by manually curating a high-quality 1,000-sample Chinese spelling correction dataset to address data quality issues.

Proposed method

• Conducted few-shot learning experiments using GPT-3.5-Turbo for text correction tasks, including CSC, CTC, and GEC, with prompt-based instruction.
• Created a new manually curated dataset of 1,000 Chinese spelling correction samples to mitigate data quality issues present in existing benchmarks.
• Evaluated model outputs using both human evaluation and self-evaluation via GPT-3.5-Turbo to ensure reliability and consistency.
• Introduced controlled noise—such as spelling errors, word insertions, and deletions—into benchmark datasets (e.g., AGIEval, Gaokao Bench, sentiment classification, machine translation) to test LLM robustness.
• Analyzed correction outputs for over-correction and under-correction, particularly focusing on semantic drift, structural optimization, and inappropriate answer generation for questions.
• Compared model outputs against gold-standard sentences to quantify accuracy, grammatical correctness, and semantic preservation.

Experimental results

Research questions

• RQ1 Can large language models like GPT-3.5-Turbo perform text correction effectively across multiple languages and error types, including spelling, grammar, and expression errors?
• RQ2 To what extent do LLMs over-correct or under-correct by altering sentence meaning or structure during correction tasks?
• RQ3 How tolerant are LLMs to noisy input data, especially when errors involve spelling versus structural modifications like word insertion or deletion?
• RQ4 Does the quality of training data significantly affect the rate of false corrections in LLM-based text correction systems?
• RQ5 Can LLMs maintain high performance on downstream NLP tasks when the input contains a certain proportion of erroneous text?

Key findings

• GPT-3.5-Turbo demonstrated strong performance in text correction tasks, producing grammatically correct and semantically accurate outputs, especially in Chinese spelling correction (CSC), with minimal false corrections after dataset quality improvements.
• The model frequently optimized sentence expression and structure, leading to over-correction in complex tasks like CTC and GEC, where it replaced less common terms with more standard equivalents, sometimes altering meaning.
• Over-correction was particularly evident in cases involving semantic repetition (e.g., '比较更' → '更') or homophonic/phonetically similar errors, where the model preferred common expressions over precise ones.
• When introducing 5% spelling errors into input data, LLM performance declined only slightly, indicating strong tolerance to low-level noise.
• Performance dropped significantly when noise involved structural changes such as word insertion or deletion, which disrupted sentence coherence and meaning.
• In 40% of cases involving question-like inputs, GPT-3.5-Turbo answered the question instead of correcting it, indicating a tendency to prioritize response generation over strict correction fidelity.