[Paper Review] Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speech

Huang Fan, Haewoon Kwak+1 more|arXiv (Cornell University)|Sep 11, 2022

Topic Modeling4 citations

TL;DR

This paper introduces Chain of Explanation (CoE) prompting, a novel method that enhances natural language explanation (NLE) quality for implicit hate speech by incorporating heuristic words, hateful intent demonstration, and target group identification. The approach boosts BLEU-1 scores from 44.0 to 62.3 and achieves strong human evaluation scores, with generated NLEs showing high informativeness (4.48/7) and clarity (4.34/7), outperforming baseline models and aligning closely with human-written explanations.

ABSTRACT

Recent studies have exploited advanced generative language models to generate Natural Language Explanations (NLE) for why a certain text could be hateful. We propose the Chain of Explanation (CoE) Prompting method, using the heuristic words and target group, to generate high-quality NLE for implicit hate speech. We improved the BLUE score from 44.0 to 62.3 for NLE generation by providing accurate target information. We then evaluate the quality of generated NLE using various automatic metrics and human annotations of informativeness and clarity scores.

Motivation & Objective

• To address the lack of high-quality natural language explanations (NLEs) for implicit hate speech, which are harder to detect than explicit forms.
• To explore the potential of prompt engineering in improving NLE generation using pre-trained language models (PLMs).
• To evaluate NLE quality beyond standard automatic metrics by incorporating human annotations for informativeness and clarity.
• To investigate the correlation between automatic metrics and human judgment in NLE evaluation.
• To propose a prompting framework that enhances faithfulness and interpretability of NLEs for implicit hate speech detection.

Proposed method

• Proposes a Chain of Explanation (CoE) prompting framework that structures input with heuristic words, hateful intent demonstration, and target group identification.
• Uses a sequence-to-sequence input format: [STR] + H_text + tweet + [SEP] + D_hate + [SEP] + D_target + [SEP] + H_NLE + generated NLE + [END].
• Incorporates structured prompting with: 'Given Text: ', 'Is the text hateful? Yes', 'The target group is: {target}', and 'It is hateful because:' to guide explanation generation.
• Trains and evaluates multiple generative language models (e.g., BART, T5) using the CoE prompt on the LatentHatred dataset.
• Employs both automatic metrics (BLEU, ROUGE, BERTScore, BLEURT, SARI, NUBIA) and human evaluation for comprehensive assessment.
• Uses three experienced research assistants for human annotation of informativeness and clarity, with inter-rater reliability (Krippendorff’s alpha) of 0.35 after filtering ambiguous cases.

Experimental results

Research questions

• RQ1 Can prompting methods significantly improve the quality of natural language explanations (NLEs) for implicit hate speech?
• RQ2 How does incorporating target group information affect NLE quality and automatic metric scores?
• RQ3 To what extent do automatic metrics like BLEU, ROUGE, and BERTScore correlate with human judgments of informativeness and clarity?
• RQ4 How does the CoE prompting method compare to baseline autoregressive generation in terms of explanation quality and faithfulness?
• RQ5 What is the impact of structured prompting on reducing ambiguity and improving the interpretability of model-generated explanations?

Key findings

• The CoE prompting method increases the BLEU-1 score from 44.0 (baseline) to 62.3, demonstrating a significant improvement in n-gram overlap with reference explanations.
• Including the target group in the prompt leads to a 15.5-point drop in BLEU-1 score when removed, confirming its critical role in explanation quality.
• Human-annotated informativeness and clarity scores for the best model’s NLEs are 4.48/7 and 4.34/7, respectively, showing strong alignment with human-written explanations (5.20 and 4.53).
• BLEURT, BERTScore, and NUBIA show the strongest Spearman correlations with human judgments (r = 0.50, 0.34, and 0.48, respectively), outperforming traditional metrics like ROUGE-L and SARI.
• SARI shows the weakest correlation (0.051 with clarity), indicating limited reliability for NLE evaluation.
• The correlation between automatic metrics and human ratings varies: BLEU-1 and ROUGE-L correlate more strongly with informativeness (r = 0.31 and 0.32) than with clarity (r = 0.15 and 0.18), highlighting metric limitations in capturing semantic clarity.