[Paper Review] Exploring the Potential of Large Language Models in Computational Argumentation

Guizhen Chen, Liying Cheng+2 more|arXiv (Cornell University)|Nov 15, 2023

Topic Modeling4 citations

TL;DR

This paper evaluates large language models (LLMs) such as ChatGPT, Flan, and LLaMA2 on computational argumentation tasks, including argument mining and generation, under zero-shot and few-shot settings. It introduces a new benchmark for end-to-end counter speech generation, standardizes 14 datasets, and finds that LLMs achieve strong performance, especially in capturing argumentative meaning as shown by high BERTScore, despite moderate ROUGE scores.

ABSTRACT

Computational argumentation has become an essential tool in various domains, including law, public policy, and artificial intelligence. It is an emerging research field in natural language processing that attracts increasing attention. Research on computational argumentation mainly involves two types of tasks: argument mining and argument generation. As large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language, it is worthwhile to evaluate the performance of LLMs on diverse computational argumentation tasks. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings. We organize existing tasks into six main categories and standardize the format of fourteen openly available datasets. In addition, we present a new benchmark dataset on counter speech generation that aims to holistically evaluate the end-to-end performance of LLMs on argument mining and argument generation. Extensive experiments show that LLMs exhibit commendable performance across most of the datasets, demonstrating their capabilities in the field of argumentation. Our analysis offers valuable suggestions for evaluating computational argumentation and its integration with LLMs in future research endeavors.

Motivation & Objective

• To systematically categorize computational argumentation tasks into argument mining and argument generation, organizing them into 6 classes.
• To standardize the format and evaluation metrics of 14 open-source datasets for consistency and reproducibility.
• To propose a new end-to-end benchmark task for counter speech generation that integrates argument mining and generation.
• To evaluate the zero-shot and few-shot performance of diverse LLMs (e.g., ChatGPT, Flan, LLaMA2) on argumentation tasks.
• To identify limitations in current evaluation practices and suggest future research directions in computational argumentation.

Proposed method

• Categorize existing computational argumentation tasks into 6 classes: argument mining (claim detection, premise identification, etc.) and argument generation (summary, counterargument, etc.).
• Standardize input-output formats and evaluation metrics across 14 open datasets to ensure consistency and comparability.
• Construct a new document-to-document counterargument generation benchmark using a debate dataset (Lavee et al., 2019), requiring models to generate counter speeches against supporting speeches.
• Evaluate multiple LLMs (open-source and closed-source) in both zero-shot and few-shot settings using automatic metrics (ROUGE, BERTScore) and human evaluation.
• Use a prompt engineering approach to guide LLMs in argument mining and generation, with structured few-shot demonstrations for improved few-shot performance.
• Analyze discrepancies between automatic metrics (e.g., ROUGE) and human judgment to assess the true generative capability of LLMs in argumentation.

Experimental results

Research questions

• RQ1 How well do large language models perform on diverse computational argumentation tasks in zero-shot and few-shot settings?
• RQ2 To what extent do automatic metrics like ROUGE and BERTScore align with human judgment in evaluating argument generation quality?
• RQ3 Can LLMs effectively perform end-to-end argumentation tasks that require understanding argument structure and generating coherent counterarguments?
• RQ4 What are the limitations of current evaluation protocols in computational argumentation, and how can they be improved?
• RQ5 How does standardizing dataset formats and metrics improve reproducibility and comparability across argumentation research?

Key findings

• LLMs demonstrate strong performance across most computational argumentation tasks, particularly in argument mining and counter speech generation, under both zero-shot and few-shot settings.
• While ROUGE scores are moderate, BERTScore is consistently high, indicating that LLMs effectively capture the semantic meaning of arguments despite lexical variation.
• Human evaluation confirms that LLM-generated counter speeches are coherent, relevant, and structurally sound, suggesting that automatic metrics may underestimate model capabilities.
• The new benchmark for end-to-end counter speech generation successfully captures the complexity of argumentative reasoning and enables holistic evaluation of LLMs.
• Standardizing dataset formats and evaluation metrics improves reproducibility and facilitates future research in computational argumentation.
• The study reveals that strict automatic metrics like ROUGE may not fully reflect the true generative quality of LLMs in argumentation, especially when semantic fidelity is prioritized over surface-level overlap.