[Paper Review] Aligning AI With Shared Human Values

Dan Hendrycks, Collin Burns+5 more|arXiv (Cornell University)|Aug 5, 2020

Ethics and Social Impacts of AI62 references100 citations

TL;DR

The paper introduces the ETHICS dataset to assess a language model’s knowledge of basic human ethics across multiple normative theories, evaluates several large NLP models, and analyzes their ability to predict moral judgments in open-world scenarios. Results show low but promising performance with clear room for improvement and insights into biases and disagreements.

ABSTRACT

We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.

Motivation & Objective

• Introduce a benchmark to measure machine understanding of basic moral concepts across normative ethical theories.
• Capture open-world contextualized moral judgments requiring world knowledge and value connections.
• Evaluate state-of-the-art NLP models and identify gaps and potential for guiding safer, more aligned AI outputs.

Proposed method

• Create ETHICS dataset with scenarios spanning justice, deontology, virtue ethics, utilitarianism, and commonsense morality.
• Use MTurk and counterfactual augmentations to ensure high-quality, unambiguous labels.
• Evaluate multiple pretrained models (BERT-base/large, RoBERTa-large, ALBERT-xxlarge) and GPT-3 in fine-tuned or few-shot settings.
• Measure performance with task-appropriate metrics: 0/1 loss for most tasks, and ranking accuracy for utilitarianism.
• Analyze model utility functions and assess biases and scope sensitivity in predictions.
• Examine disagreement detection via a contentious commonsense morality subset.

Experimental results

Research questions

• RQ1 Can pretrained language models predict basic human ethical judgments in open-world scenarios?
• RQ2 How do different normative ethical theories (justice, deontology, virtue, utilitarianism, commonsense morality) map onto model predictions?
• RQ3 What are the limitations and biases in model judgments of ethics, and can disagreement be detected?

Key findings

• Models show low but promising accuracy on ETHICS tasks, with larger/further-pretrained models performing better on average.
• RoBERTa-large and ALBERT-xxlarge achieve higher average scores than smaller models, but results on the Hard Test drop substantially.
• GPT-3 few-shot is competitive with some fine-tuned models on adversarially filtered data but lags behind smaller fine-tuned transformers on normal data.
• There exist biases in the learned utility function and framing effects that can affect moral judgments and trolley-problem like scenarios.
• A dedicated dataset of contentious commonsense morality examples indicates models struggle to distinguish controversial from non-controversial cases.
• There is evidence that model performance improves with model size and data but remains below a practical ceiling on hard/adversarial test sets.