[Paper Review] CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation

Yujie Shao, Xinrong Yao+6 more|arXiv (Cornell University)|Feb 20, 2024

Language, Metaphor, and CognitionPsychology3 citations

TL;DR

This paper introduces CMDAG, a large-scale Chinese metaphor dataset with 28,000 sentences annotated for tenor, vehicle, and ground (喻意), emphasizing conceptual mapping. By integrating ground-based Chain-of-Thought (CoT) prompting, the method boosts generation of fluent, creative, and contextually appropriate metaphors in Chinese, significantly improving model performance on metaphor generation tasks compared to conventional approaches.

ABSTRACT

Metaphor is a prominent linguistic device in human language and literature, as they add color, imagery, and emphasis to enhance effective communication. This paper introduces a large-scale high quality annotated Chinese Metaphor Corpus, which comprises around 28K sentences drawn from a diverse range of Chinese literary sources, such as poems, prose, song lyrics, etc. To ensure the accuracy and consistency of our annotations, we introduce a comprehensive set of guidelines. These guidelines address the facets of metaphor annotation, including identifying tenors, vehicles, and grounds to handling the complexities of similes, personifications, juxtapositions, and hyperboles. Breaking tradition, our approach to metaphor generation emphasizes grounds and their distinct features rather than the conventional combination of tenors and vehicles. By integrating "ground" as a CoT (Chain of Thoughts) input, we are able to generate metaphors that resonate more with real-world intuition. We test generative models such as Belle, Baichuan, and Chinese-alpaca-33B using our annotated corpus. These models are able to generate creative and fluent metaphor sentences more frequently induced by selected samples from our dataset, demonstrating the value of our corpus for Chinese metaphor research. The code is available in https://github.com/JasonShao55/Chinese_Metaphor_Explanation.

Motivation & Objective

• To address the lack of high-quality, structured Chinese metaphor datasets that include conceptual grounds (喻意) for improved metaphor generation.
• To develop a rigorous annotation framework that ensures consistency and accuracy in identifying tenors, vehicles, and grounds in metaphors.
• To explore the effectiveness of Chain-of-Thought (CoT) prompting using grounds as intermediate reasoning steps to guide metaphor generation.
• To evaluate how ground-based CoT influences the fluency, creativity, and authenticity of generated metaphors in Chinese.
• To demonstrate the utility of CMDAG for training and evaluating large language models in generating human-like metaphors.

Proposed method

• Annotate 27,989 metaphorical sentences from diverse Chinese literary sources (poetry, prose, lyrics) with explicit tenor (本体), vehicle (喻体), and ground (喻意) components.
• Design a comprehensive annotation guideline to standardize identification of metaphors, similes, personifications, and hyperboles, with special attention to ground features.
• Propose a novel CoT-based prompting strategy where grounds are used as reasoning steps to guide LLMs in generating metaphors from tenor-vehicle pairs or tenor-ground pairs.
• Conduct two evaluation setups: (1) given tenor and vehicle, generate ground via CoT; (2) given tenor and ground, generate vehicle via CoT.
• Fine-tune and evaluate open-source LLMs (Belle, Baichuan, Chinese-alpaca-33B) using the CMDAG dataset under CoT prompting.
• Apply human evaluation with criteria including clarity, creativity, and authenticity to assess model outputs, and analyze correlations between evaluation dimensions.

Experimental results

Research questions

• RQ1 How does incorporating ground (喻意) as a Chain-of-Thought (CoT) component improve the quality of generated Chinese metaphors?
• RQ2 To what extent does ground-based CoT enhance fluency, creativity, and authenticity in metaphor generation compared to standard prompting?
• RQ3 How do different large language models (LLMs) perform on Chinese metaphor generation when fine-tuned with CMDAG and CoT?
• RQ4 What is the relationship between evaluation criteria such as clarity, creativity, and authenticity in human judgments of generated metaphors?
• RQ5 Can a dataset with explicit ground annotations significantly improve the performance of LLMs in generating contextually appropriate and conceptually coherent metaphors?

Key findings

• Grounds-based CoT significantly increases the proportion of reasonable metaphorical sentences generated by LLMs, especially when compared to standard prompting without ground reasoning.
• LLMs with more extensive Chinese pretraining data (e.g., Baichuan) outperform general multilingual models like GPT-3.5 and GPT-4 in Chinese metaphor generation, indicating domain-specific pretraining improves performance.
• Belle generates significantly fewer reasonable metaphors than GPT-4 and GPT-3.5, highlighting that model architecture and pretraining data quality are critical for this task.
• Human evaluators prioritize creativity over authenticity in metaphor evaluation, and clarity becomes more important in CoT-based settings, suggesting a shift in human judgment when reasoning steps are involved.
• The Pearson correlation between evaluation criteria shows that clarity and creativity are moderately correlated (r=0.28), while creativity and authenticity are highly correlated (r=0.71), indicating that creative metaphors are more likely to be judged as authentic.
• The study reveals that ground-based CoT may slightly reduce model exploration of novel vehicles and grounds, suggesting a trade-off between coherence and originality in metaphor generation.