[Paper Review] Building Interpretable Models for Moral Decision-Making

Mayank Goel, Aritra Das+1 more|arXiv (Cornell University)|Feb 3, 2026

Psychology of Moral and Emotional Judgment0 citations

TL;DR

The authors design a compact 2-layer transformer tailored for structured trolley-problem scenarios, achieving 77% validation accuracy on Moral Machine data and enabling detailed mechanistic interpretability analyses of moral reasoning within a small model.

ABSTRACT

We build a custom transformer model to study how neural networks make moral decisions on trolley-style dilemmas. The model processes structured scenarios using embeddings that encode who is affected, how many people, and which outcome they belong to. Our 2-layer architecture achieves 77% accuracy on Moral Machine data while remaining small enough for detailed analysis. We use different interpretability techniques to uncover how moral reasoning distributes across the network, demonstrating that biases localize to distinct computational stages among other findings.

Motivation & Objective

• Demonstrate that a small, purpose-built model can learn general moral principles from structured trolley-problem scenarios.
• Show that interpretability techniques can localize moral biases to distinct computational stages in a transformer.
• Provide a mechanistic analysis of how character types, counts, and team membership influence moral judgments.
• Propose and validate an architecture that encodes who is affected, how many are affected, and which side they belong to.

Proposed method

• Design a 2-layer transformer with 64-dim embeddings and 2 attention heads.
• Use compositional embeddings per character: [char embedding; cardinality embedding; team embedding].
• Prepend a [CLS] token and apply a standard transformer encoder to a 46-token input (23 per outcome).
• Train a two-layer MLP head on the CLS representation to output a scalar logit for outcome preference.
• Enforce side-invariance via a symmetrization at inference using the average of forward and reverse orderings.
• Evaluate across architecture variants to select a balance of accuracy and interpretability.
• Apply causal intervention (DoWhy) to estimate average treatment effects of character types.
• Perform layer-wise attribution to map biases to specific layers and heads.
• Use circuit probing to identify sparse subnetworks enabling the final moral score.
• Compute local relevance with gradient-weighted attention across tokens and average remapped scores for symmetry.

Figure 1: Overview of the paper

Experimental results

Research questions

• RQ1 Can a compact transformer learn general moral principles from structured trolley-problem data?
• RQ2 Where do moral biases localize within a shallow transformer (which layers/heads)?
• RQ3 To what extent can causal interventions, layer-wise attributions, and circuit probing reveal the internal mechanics of moral judgment?
• RQ4 How does the model's architecture encode who is affected, how many are affected, and team assignment to drive decisions?

Key findings

• The final model with d=64, H=2, L=2 achieves 77.1% validation accuracy on held-out Moral Machine scenarios.
• Causal intervention shows a clear moral hierarchy among characters, with Pregnant and Stroller providing positive influence and Criminal providing negative influence.
• Layer-wise bias localization reveals legality bias mainly in Layer 0, while species bias emerges mainly in Layer 1, indicating functional specialization across layers.
• Circuit probing identifies a sparse subnetwork (45 of 256 neurons) in the Layer 1 MLP that causally contributes to the final score, with an ablation drop of 0.012 in accuracy.
• Local relevance analysis attributes substantial decision impact to key tokens like Criminal (0.27 relative relevance) and contextual/role-based tokens, with demographic tokens contributing less.

Figure 2: Causal influence of character types on model decisions, estimated via DoWhy backdoor adjustment controlling for group sizes. Blue bars indicate positive causal effects (model favors outcomes containing these characters); red bars indicate negative effects.