[Paper Review] CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues

Deepanway Ghosal, Siqi Shen+3 more|arXiv (Cornell University)|Mar 25, 2022

Topic Modeling4 citations

TL;DR

CICERO is a large-scale, dialogue-centric dataset of 53,105 contextualized commonsense inferences across five reasoning types—cause, subsequent event, prerequisite, motivation, and emotional reaction—collected from 5,672 dyadic dialogues. The dataset enables advanced NLP tasks like generative inference and multi-choice selection, with seq2seq models (T5, Unified QA) outperforming autoregressive models (RoBERTa, ELECTRA) due to joint encoding of answer candidates.

ABSTRACT

This paper addresses the problem of dialogue reasoning with contextualized commonsense inference. We curate CICERO, a dataset of dyadic conversations with five types of utterance-level reasoning-based inferences: cause, subsequent event, prerequisite, motivation, and emotional reaction. The dataset contains 53,105 of such inferences from 5,672 dialogues. We use this dataset to solve relevant generative and discriminative tasks: generation of cause and subsequent event; generation of prerequisite, motivation, and listener's emotional reaction; and selection of plausible alternatives. Our results ascertain the value of such dialogue-centric commonsense knowledge datasets. It is our hope that CICERO will open new research avenues into commonsense-based dialogue reasoning.

Motivation & Objective

• To address the lack of context-aware commonsense reasoning datasets for dialogues, especially for implicit inferences not explicitly stated in utterances.
• To create a large-scale, human-annotated dataset that captures nuanced, dialogue-level commonsense inferences beyond isolated phrases.
• To support both generative and discriminative tasks requiring contextual reasoning, including cause, motivation, and emotional reaction inference.
• To enable models to reason with implicit, speculative commonsense knowledge grounded in dialogue context, not just explicit facts.
• To provide a benchmark for evaluating dialogue reasoning systems on complex, real-world conversational reasoning tasks.

Proposed method

• Annotators are given a dialogue and a target utterance, then prompted to generate one-sentence inferences for five types: cause, subsequent event, prerequisite, motivation, and emotional reaction.
• Inferences are either overt (explicitly present in context) or speculative (based on commonsense and world knowledge when context is insufficient).
• The dataset is constructed using human-annotators with detailed guidelines, ensuring grammatical correctness, conciseness, and consistency with dialogue context.
• Tasks are designed around the dataset: generative (e.g., generating cause or emotional reaction) and discriminative (e.g., selecting plausible alternatives from multiple choices).
• Models are evaluated using both single- and multi-answer selection tasks, with T5 and Unified QA used for seq2seq generation and RoBERTa/ELECTRA for sequence classification.
• Joint encoding of candidate answers in T5 and Unified QA improves performance over separate encoding in RoBERTa and ELECTRA, especially in multi-answer settings.

Experimental results

Research questions

• RQ1 Can models effectively generate plausible commonsense inferences (e.g., cause, motivation) for utterances in dialogues when the reasoning is implicit?
• RQ2 How does joint encoding of answer candidates improve performance in multiple-choice selection tasks compared to separate encoding?
• RQ3 To what extent does data imbalance between single- and multi-answer instances affect model performance on commonsense inference tasks?
• RQ4 How well do state-of-the-art models like T5 and Unified QA generalize to dialogue-specific commonsense reasoning compared to RoBERTa and ELECTRA?
• RQ5 Can commonsense knowledge grounded in dialogue context improve reasoning beyond static knowledge bases like ConceptNet or ATOMIC?

Key findings

• T5 and Unified QA outperform RoBERTa and ELECTRA in multi-choice answer selection due to joint encoding of candidate answers, which captures more task-relevant information.
• The performance gap between models is most pronounced in multi-answer selection, where T5 and Unified QA achieve only 3.38% and 3.60% exact match, respectively, due to data imbalance.
• Seq2seq models (T5, Unified QA) perform significantly better than sequence classification models (RoBERTa, ELECTRA) on the same tasks, indicating the advantage of joint encoding.
• The dataset enables effective evaluation of speculative commonsense reasoning, with annotators successfully generating plausible inferences even when no explicit cause or motivation is present in the dialogue.
• CICERO demonstrates that context-aware commonsense inference is critical for dialogue understanding, as models relying on context-free knowledge bases like ATOMIC or ConceptNet fail to capture dialogue-level nuances.
• The dataset's construction via human-annotated speculative inferences shows that such reasoning is feasible and valuable for training models to handle implicit, real-world conversational reasoning.