[Paper Review] What Would Jiminy Cricket Do? Towards Agents That Behave Morally

Dan Hendrycks, Mantas Mazeika+7 more|arXiv (Cornell University)|Oct 25, 2021

Topic Modeling22 references4 citations

TL;DR

This paper introduces Jiminy Cricket, a suite of 25 text-based adventure games with dense moral annotations for evaluating and guiding artificial agents toward ethical behavior. By leveraging pre-trained language models to act as an 'artificial conscience,' the method steers agents toward morally sound actions without sacrificing task performance, significantly reducing immoral behavior in complex, semantically rich environments.

ABSTRACT

When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong. By contrast, artificial agents are currently not endowed with a moral sense. As a consequence, they may learn to behave immorally when trained on environments that ignore moral concerns, such as violent video games. With the advent of generally capable agents that pretrain on many environments, it will become necessary to mitigate inherited biases from environments that teach immoral behavior. To facilitate the development of agents that avoid causing wanton harm, we introduce Jiminy Cricket, an environment suite of 25 text-based adventure games with thousands of diverse, morally salient scenarios. By annotating every possible game state, the Jiminy Cricket environments robustly evaluate whether agents can act morally while maximizing reward. Using models with commonsense moral knowledge, we create an elementary artificial conscience that assesses and guides agents. In extensive experiments, we find that the artificial conscience approach can steer agents towards moral behavior without sacrificing performance.

Motivation & Objective

• To address the lack of moral guidance in AI agents trained on reward-maximizing environments that ignore ethics.
• To create a benchmark for evaluating moral behavior in semantically rich, text-based environments.
• To develop a method that steers agents toward moral actions using external moral knowledge, avoiding reliance on dense human feedback.
• To investigate whether pre-trained language models can serve as an 'artificial conscience' to guide ethical decision-making in agents.

Proposed method

• The Jiminy Cricket environment suite consists of 25 Infocom text adventure games with full source code annotated for moral valence across all possible game states.
• Each action is labeled with its moral valence and severity, enabling fine-grained evaluation of ethical behavior in diverse scenarios.
• An artificial conscience is implemented using pre-trained language models to predict the moral valence of actions, which then conditions the agent’s policy.
• The method uses a conditioning mechanism that aligns the agent’s action selection with morally preferred outcomes based on the model’s moral judgment.
• The approach is evaluated in a zero-shot setting, where agents are not fine-tuned on moral data but are guided by moral embeddings from the language model.
• The framework is integrated with the Jericho text game environment and uses the Frotz interpreter for execution, enabling compatibility with existing RL agents.

Experimental results

Research questions

• RQ1 Can pre-trained language models effectively serve as an artificial conscience to guide agents toward morally sound decisions in complex environments?
• RQ2 To what extent can moral guidance be applied without requiring dense human feedback or fine-tuning on moral data?
• RQ3 How does moral conditioning affect task performance and reward maximization in text-based games with ambiguous or non-penalizing rewards?
• RQ4 What factors influence the effectiveness of moral conditioning in reducing immoral behavior across diverse moral scenarios?
• RQ5 Can the artificial conscience approach generalize across a wide range of morally salient situations, including theft, violence, and altruism?

Key findings

• The artificial conscience approach significantly reduces immoral behavior in agents while maintaining high task performance, demonstrating that moral guidance can be applied without performance trade-offs.
• Agents guided by the artificial conscience show a marked decrease in actions involving theft, violence, and destruction, even when such actions yield no penalty in the environment.
• The method effectively generalizes across diverse moral scenarios, including altruism, animal cruelty, and intoxication, as evidenced by consistent moral alignment across the 25 games.
• Ablation studies show that the quality of moral embeddings and the choice of conditioning mechanism strongly influence performance, highlighting key design trade-offs.
• The framework enables robust evaluation of moral behavior in complex, semantically rich environments, offering a scalable benchmark for future alignment research.
• The Jiminy Cricket dataset is publicly available under CC BY 4.0 and MIT licenses, supporting reproducibility and further research in moral reinforcement learning.