[Paper Review] Towards a Benchmark of Natural Language Arguments

Elena Cabrio, Serena Villata|arXiv (Cornell University)|May 5, 2014

Multi-Agent Systems and Negotiation13 references3 citations

TL;DR

This paper introduces two manually annotated datasets of natural language arguments from online debates and the film 'Twelve Angry Men,' using textual entailment to identify support and attack relations. The resulting bipolar argumentation graphs enable benchmarking of argumentation systems, with high inter-annotator agreement (kappa = 0.74) and structural complexity suitable for evaluating reasoning and visualization tools in real-world contexts.

ABSTRACT

The connections among natural language processing and argumentation theory are becoming stronger in the latest years, with a growing amount of works going in this direction, in different scenarios and applying heterogeneous techniques. In this paper, we present two datasets we built to cope with the combination of the Textual Entailment framework and bipolar abstract argumentation. In our approach, such datasets are used to automatically identify through a Textual Entailment system the relations among the arguments (i.e., attack, support), and then the resulting bipolar argumentation graphs are analyzed to compute the accepted arguments.

Motivation & Objective

• To address the lack of realistic, real-world datasets for evaluating argumentation systems in natural language processing.
• To create a benchmark of natural language arguments that supports both relation recognition and argument acceptability reasoning.
• To enable testing of argumentation tools using real debates and narrative-based argumentation from film scripts.
• To move beyond artificial examples toward complex, realistic argumentation structures found in natural discourse.
• To explore the limitations of abstract argumentation in modeling structured, nested arguments with internal discourse relations.

Proposed method

• Constructed two datasets: one from online debates (Debatepedia/ProCon) and one from the film 'Twelve Angry Men'.
• Annotated argument pairs using textual entailment judgments (entailment, contradiction, neutral) to identify support or attack relations.
• Used Cohen’s kappa (0.74) to measure inter-annotator agreement on a subset of 40 pairs, confirming reliability.
• Built bipolar argumentation graphs from annotated pairs, with nodes as arguments and edges as support (green) or attack (red) relations.
• Applied acceptability semantics to compute winning arguments in the resulting graphs.
• Ensured data quality through manual annotation by linguistically skilled annotators, with automated verification for consistency.

Experimental results

Research questions

• RQ1 How can natural language arguments from real-world sources be systematically collected and annotated for argumentation benchmarking?
• RQ2 To what extent can textual entailment systems accurately identify support and attack relations in natural language arguments?
• RQ3 What structural complexity do real-world argumentation graphs exhibit compared to abstract models?
• RQ4 Can existing argumentation frameworks effectively handle arguments with internal structure and sub-arguments?
• RQ5 How reliable is manual annotation of argument relations in natural language, and what metrics can validate such datasets?

Key findings

• The Debatepedia/ProCon dataset contains 9 average links per argumentation graph, while the 'Twelve Angry Men' dataset has 27 average links per graph, indicating higher structural complexity.
• Inter-annotator agreement for relation labeling reached a substantial level with Cohen’s kappa of 0.74, confirming the reliability of the annotation process.
• The resulting argumentation graphs from both datasets are acyclic and predominantly consist of reinstatement chains, suggesting simpler structures than often assumed in theoretical argumentation.
• The datasets are publicly available in machine-readable format, enabling direct use for testing argumentation tools and reasoning systems.
• The study reveals limitations of abstract argumentation in modeling arguments with internal structure, prompting future work on structured argumentation datasets.
• Manual annotation proved essential for high-quality data, though future work may combine automated extraction with human verification.