[论文解读] SAD: A Large-Scale Strategic Argumentative Dialogue Dataset

Yongkang Liu, Jiayang Yu+7 more|arXiv (Cornell University)|Jan 12, 2026

Topic Modeling被引用 0

一句话总结

tldr: SAD 是一个大规模、具有策略感知的多轮论证对话数据集，包含 392,822 个样本（超过 722k 句子）并附注立场与五种论证策略，用于研究策略条件生成及评估模型的说服力。

ABSTRACT

Argumentation generation has attracted substantial research interest due to its central role in human reasoning and decision-making. However, most existing argumentative corpora focus on non-interactive, single-turn settings, either generating arguments from a given topic or refuting an existing argument. In practice, however, argumentation is often realized as multi-turn dialogue, where speakers defend their stances and employ diverse argumentative strategies to strengthen persuasiveness. To support deeper modeling of argumentation dialogue, we present the first large-scale extbf{S}trategic extbf{A}rgumentative extbf{D}ialogue dataset, SAD, consisting of 392,822 examples. Grounded in argumentation theories, we annotate each utterance with five strategy types, allowing multiple strategies per utterance. Unlike prior datasets, SAD requires models to generate contextually appropriate arguments conditioned on the dialogue history, a specified stance on the topic, and targeted argumentation strategies. We further benchmark a range of pretrained generative models on SAD and present in-depth analysis of strategy usage patterns in argumentation.

研究动机与目标

• Motivate the study of real-world, interactive argumentation beyond single-turn setups.
• Create a large, high-quality dataset of multi-turn argumentative dialogues annotated with stance and five strategies.
• Ground the dataset in theory and real-world data from ChangeMyView to enable strategy-conditioned generation.
• Propose a strategy-conditioned generation task: P(A | History, Stance, Topic, Strategy).
• Develop automatic and human evaluations to assess fluency, coherence, topicality, and persuasiveness, and benchmark LLMs.

提出的方法

• Construct a large-scale dataset SAD from ChangeMyView discussions (CMV) with 392,822 dialogue examples and 722,812 utterances across 20,619 topics.
• Annotate each utterance with stance (support vs. opposition) via majority voting among five workers (Fleiss’ kappa = 0.78).
• Annotate utterances with five strategy types (Question, Causality, Example, Analogy, Statement) with the possibility of multiple labels per utterance.
• Implement quality control for strategy annotations, including pre-annotation practice, expert revision, and random sampling consistency checks (>97.2% agreement with at least one annotator, >91.0% with at least two).
• Formulate and evaluate a strategy-conditioned generation task: P(A | History, Stance, Topic, Strategy).
• Develop an automatic persuasiveness evaluator based on Like counts and perform human evaluation across fluency, coherence, relevance, and persuasiveness.

实验结果

研究问题

• RQ1 How do strategy annotations influence the quality and characteristics of multi-turn argumentative generation?
• RQ2 Does incorporating explicit argument strategies improve generation fluency, coherence, topicality, and persuasiveness across models?
• RQ3 How do open-source and closed-source models differ in leveraging strategy information, and what is the impact of fine-tuning and optimization strategies (SFT vs. DPO) on performance?
• RQ4 What are the empirical patterns of strategy usage and transitions throughout multi-turn debates in SAD?

主要发现

• SAD comprises 392,822 dialogue examples and 722,812 utterances over 20,619 topics, indicating substantial scale and topical diversity.
• Five strategies are annotated per utterance (Question, Causality, Example, Analogy, Statement) with the possibility of multiple strategies per utterance, and high annotation reliability is reported.
• Explicit strategy guidance improves generation quality in terms of relevance, coherence, and fluency across multiple models and evaluation setups.
• Human evaluation shows strategy-informed generation yields gains in relevance and persuasiveness, with larger improvements when using strategy cues and fine-tuning.
• Automatic evaluation with a GPT-4.1-based persuasiveness evaluator demonstrates consistent gains in multi-dimensional argumentative quality when strategies are used, and fine-tuning (DPO) generally outperforms SFT for persuasive and coherent responses.
• Open-source models gain modest persuasiveness improvements with strategies, while closed-source models show larger gains, highlighting differences in leveraging argumentative strategies.