[Paper Review] FLUTE: Figurative Language Understanding through Textual Explanations

Tuhin Chakrabarty, Arkadiy Saakyan+2 more|arXiv (Cornell University)|May 24, 2022

Natural Language Processing Techniques7 citations

TL;DR

FLUTE is a new 9,000-sample benchmark for figurative language understanding that pairs literal and figurative sentence pairs with entailment/contradiction labels and free-form textual explanations. It uses a model-in-the-loop framework combining GPT-3 and human annotators (crowdworkers and experts) to scale high-quality data collection, and shows that fine-tuning T5 on FLUTE yields significantly better explanations than on e-SNLI, advancing the goal of building models that understand figurative language for the right reasons.

ABSTRACT

Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a. natural language inference, or NLI). However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.

Motivation & Objective

• To address the lack of explanation-based benchmarks for figurative language understanding, which hinders assessment of whether models generalize beyond spurious correlations.
• To develop a scalable, high-quality dataset of figurative NLI instances with natural language explanations to support genuine model interpretability.
• To evaluate whether training on explanation-anchored figurative language data improves model performance and explanation quality compared to existing NLI benchmarks.
• To demonstrate the effectiveness of a model-in-the-loop framework using GPT-3 and human-in-the-loop validation for constructing complex linguistic datasets.

Proposed method

• A model-in-the-loop framework generates diverse literal and figurative sentence pairs using GPT-3 with few-shot prompting, reducing manual effort.
• Crowdworkers perform minimal edits to convert literal sentences into sarcastic forms, preserving semantic coherence.
• Expert annotators validate and refine GPT-3 outputs for accuracy, consistency, and linguistic quality.
• Each instance includes a natural language explanation of the entailment/contradiction decision, grounded in the figurative expression.
• The dataset is structured as <literal, figurative> sentence pairs with labels (entail/contradict) and explanations, covering sarcasm, similes, metaphors, and idioms.
• A T5 model is fine-tuned jointly on label prediction and explanation generation using both e-SNLI and FLUTE data for comparison.

Experimental results

Research questions

• RQ1 Can a model-in-the-loop framework effectively scale the creation of high-quality, explanation-anchored datasets for figurative language?
• RQ2 Does fine-tuning on FLUTE lead to better explanation quality than fine-tuning on e-SNLI for figurative language understanding?
• RQ3 To what extent do models trained on FLUTE demonstrate more robust and faithful reasoning about figurative expressions compared to baseline models?
• RQ4 How do explanations generated by models fine-tuned on FLUTE compare to those from models trained on general NLI data in terms of coherence and relevance?

Key findings

• The T5 model fine-tuned on FLUTE produces significantly higher-quality textual explanations than the same model fine-tuned on e-SNLI, as measured by automatic and human evaluation.
• FLUTE’s inclusion of explanations tied directly to figurative expressions enables more reliable assessment of model reasoning than standard NLI benchmarks.
• The model-in-the-loop approach combining GPT-3 and expert-validated human input successfully scales the creation of complex, diverse figurative language instances with high quality.
• The dataset demonstrates that models trained on explanation-anchored figurative language data are better equipped to handle out-of-distribution and adversarial examples.
• FLUTE outperforms existing benchmarks like Big-bench and IMPLI by ensuring non-entailment examples contradict the figurative meaning, not just the literal part of the sentence.
• Human evaluation confirms that explanations generated by the FLUTE-fine-tuned model are more faithful and contextually relevant than those from models trained on general NLI data.