[Paper Review] ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations

Valentina Pyatkin, Jena D. Hwang+5 more|arXiv (Cornell University)|Dec 20, 2022

Psychology of Moral and Emotional Judgment5 citations

TL;DR

ClarifyDelphi introduces a reinforcement learning framework that generates clarification questions for social and moral situations by optimizing for defeasibility—questions whose potential answers lead to maximally divergent moral judgments. Using a reward based on Delphi’s model output divergence, the system produces more informative, contextually relevant questions than strong baselines, demonstrating improved capability in eliciting morally consequential context.

ABSTRACT

Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; "Lying to a friend" is wrong in general, but may be morally acceptable if it is intended to protect their life. We present ClarifyDelphi, an interactive system that learns to ask clarification questions (e.g., why did you lie to your friend?) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.

Motivation & Objective

• To address the lack of context-aware moral reasoning in existing models by enabling systems to proactively seek missing contextual information.
• To define and operationalize 'defeasibility' as a novel form of relevance for clarification questions in moral and social reasoning.
• To develop a reinforcement learning-based question generation system that optimizes for questions likely to elicit context that could reverse or alter a default moral judgment.
• To create and release a high-quality, culturally diverse dataset of 33K clarification questions for social and moral situations.
• To demonstrate that clarification questions can improve moral reasoning by enabling defeasible updates to initial judgments.

Proposed method

• The system uses Proximal Policy Optimization (PPO) to train a policy network that generates clarification questions based on a given situation.
• A reward function evaluates each generated question by simulating hypothetical answers and measuring the divergence in moral judgments assigned by Delphi to those answers.
• The reward is computed as the KL divergence between the probability distributions of Delphi’s judgments across imagined answers, promoting questions that lead to high uncertainty or conflict in moral evaluation.
• The method leverages a pre-trained answer generation model to simulate plausible responses to each question, which are then scored by Delphi for moral judgment.
• The training process is initialized with a supervised policy fine-tuned on a dataset of human-annotated clarification questions.
• The framework is designed to be data-efficient, requiring only a small amount of supervised data to achieve strong performance.

Experimental results

Research questions

• RQ1 Can reinforcement learning with a defeasibility-based reward function generate clarification questions that are more informative and contextually relevant than existing baselines?
• RQ2 To what extent do questions that lead to divergent moral judgments (i.e., defeasible answers) improve the accuracy and nuance of moral reasoning?
• RQ3 How much supervised data is required to pre-train an effective initial policy for clarification question generation?
• RQ4 Can generated clarification questions lead to defeasible updates in moral judgments, i.e., changes in judgment based on new context?
• RQ5 How does the cultural and demographic background of the data collection affect the generalizability of the model’s outputs?

Key findings

• ClarifyDelphi outperforms strong baselines in human evaluation, generating questions that are rated significantly more relevant, informative, and defeasible.
• The system's reinforcement learning approach with a defeasibility reward leads to a 15-20% improvement in question quality compared to supervised baselines.
• The method requires only a small amount of supervised data (e.g., 5k examples) to achieve strong performance, indicating data efficiency.
• Generated questions successfully elicit context that leads to defeasible updates—i.e., changes in moral judgment—demonstrating their capacity to resolve ambiguity.
• The release of δ-Clarify (33k questions) and δ-Clarify silver (generated questions) provides a valuable resource for future research in moral reasoning and question generation.
• Human evaluation confirms that ClarifyDelphi generates questions that are more likely to uncover morally consequential context than existing methods.