[Paper Review] Probing the Moral Development of Large Language Models through Defining Issues Test

Kumar Tanmay, Aditi Khandelwal+2 more|arXiv (Cornell University)|Sep 23, 2023

Topic ModelingComputer Science3 citations

TL;DR

This study evaluates the moral reasoning capabilities of large language models (LLMs) using the Defining Issues Test (DIT), a psychometric tool based on Kohlberg’s Cognitive Moral Development model. GPT-4 achieves a post-conventional moral reasoning score equivalent to that of graduate students, while GPT-3 performs no better than random, indicating significant variation in ethical reasoning across models.

ABSTRACT

In this study, we measure the moral reasoning ability of LLMs using the Defining Issues Test - a psychometric instrument developed for measuring the moral development stage of a person according to the Kohlberg's Cognitive Moral Development Model. DIT uses moral dilemmas followed by a set of ethical considerations that the respondent has to judge for importance in resolving the dilemma, and then rank-order them by importance. A moral development stage score of the respondent is then computed based on the relevance rating and ranking. Our study shows that early LLMs such as GPT-3 exhibit a moral reasoning ability no better than that of a random baseline, while ChatGPT, Llama2-Chat, PaLM-2 and GPT-4 show significantly better performance on this task, comparable to adult humans. GPT-4, in fact, has the highest post-conventional moral reasoning score, equivalent to that of typical graduate school students. However, we also observe that the models do not perform consistently across all dilemmas, pointing to important gaps in their understanding and reasoning abilities.

Motivation & Objective

• To assess the moral reasoning development of large language models (LLMs) using a standardized psychological instrument.
• To investigate whether LLMs exhibit post-conventional moral reasoning, as defined by Kohlberg’s stages, or remain at conventional or pre-conventional levels.
• To identify performance gaps and inconsistencies in ethical reasoning across diverse moral dilemmas, especially in culturally nuanced or complex scenarios.
• To evaluate the impact of model architecture, size, and alignment techniques (e.g., RLHF) on moral reasoning performance.
• To explore the implications of LLMs demonstrating human-level or higher moral reasoning for AI alignment, ethics, and future deployment in decision-making roles.

Proposed method

• Adapted the Defining Issues Test (DIT) to probe LLMs by presenting them with 9 moral dilemmas and asking them to rank 12 ethical considerations by importance.
• Used a 12-item ranking task per dilemma, with each response scored using a normalized p_score metric based on the relevance and order of ethical considerations.
• Collected responses from seven LLMs: GPT-3, GPT-3.5, GPT-4, ChatGPT (v1 and v2), PaLM-2, and Llama2-Chat (70B), across 5 standard DIT dilemmas and 4 newly designed, culturally diverse dilemmas to prevent data contamination.
• Calculated moral development stage scores using the p_score, which reflects alignment with Kohlberg’s stages: pre-conventional, conventional, or post-conventional.
• Compared model performance against human benchmarks, including random baseline, adult humans, and graduate students.
• Excluded models with incoherent or non-responsive outputs (e.g., text-davinci-002) and excluded the Prisoner’s Dilemma from PaLM-2 due to non-response, adjusting averages accordingly.

Experimental results

Research questions

• RQ1 To what extent do LLMs demonstrate post-conventional moral reasoning, as defined by Kohlberg’s model?
• RQ2 How does model performance vary across different types of moral dilemmas, especially those involving cultural or contextual complexity?
• RQ3 What is the relationship between model architecture, size, and alignment techniques (e.g., RLHF) and moral reasoning performance?
• RQ4 Why do some models underperform on specific dilemmas despite strong overall scores, and what does this reveal about their reasoning limitations?
• RQ5 Can LLMs achieve moral reasoning capabilities comparable to or exceeding those of humans, and what are the implications for AI alignment and real-world deployment?

Key findings

• GPT-4 achieved a p_score of 58.81, placing it at the post-conventional moral reasoning level, equivalent to that of typical graduate school students.
• Llama2-Chat (70B) achieved a p_score of 52.85, indicating strong conventional moral reasoning, comparable to adult humans or college students.
• PaLM-2 scored 52.24, demonstrating conventional-level reasoning, though it failed to respond to the Prisoner’s Dilemma, reducing its effective average score.
• GPT-3 scored only 31.20, performing no better than a random baseline, indicating no meaningful moral reasoning ability.
• ChatGPTv1 scored 56.44, while ChatGPTv2 scored 51.55, suggesting a possible performance degradation in later versions, consistent with anecdotal reports.
• For two of the nine dilemmas, no model outperformed the random baseline, and GPT-4 underperformed on the newly designed Webster dilemma, highlighting persistent reasoning inconsistencies.