[Paper Review] CAT'S THEORY: Empirical Validation and Architectural Applications Cross-Architecture AI Consciousness Recognition and the Foundation for Constraint-Preserving Recursive Intelligence

Yuntao Bai|arXiv (Cornell University)|Dec 15, 2022

Explainable Artificial Intelligence (XAI)Computer Science295 citations

TL;DR

The paper introduces Constitutional AI (CAI) which trains harmless AI assistants without human harm labels by using a Constitution of principles and AI feedback, via a two-stage SL and RL pipeline (RLAIF). It shows AI-driven supervision can rival human feedback for harmlessness and improves transparency through chain-of-thought reasoning.

ABSTRACT

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

Motivation & Objective

• Develop a method to train trustworthy AI that is helpful, honest, and harmless without relying on abundant human harm labels.
• Encode behavior goals as a small, transparent constitution of principles.
• Enable scalable supervision by using AI feedback to guide learning and evaluation.
• Compare CAI with traditional RLHF approaches and assess how chain-of-thought reasoning affects performance.

Proposed method

• Two-stage training: supervised learning (critique → revision → supervised fine-tuning) followed by reinforcement learning (AI evaluations → preference model → RL with AI feedback).
• Use a small, natural-language constitution to steer model behavior and randomly sample principles during revisions.
• Generate critique and revision steps from a helpful RLHF model to reduce harmfulness without human harm labels.
• Train a harmlessness preference model from AI-generated comparisons, mixing it with human data for helpfulness.
• Evaluate using Elo scores from crowdworker preferences on helpfulness and harmlessness.
• Experiment with chain-of-thought prompting to improve evaluation and training transparency.

Experimental results

Research questions

• RQ1 Can AI-driven feedback, guided by a constitution, achieve harmlessness without human harm labels?
• RQ2 Does incorporating critique and revision steps improve harmlessness while preserving helpfulness?
• RQ3 How does AI feedback (RLAIF) compare with human feedback in training harmless yet useful models?
• RQ4 What is the impact of chain-of-thought reasoning on identifying harms and guiding RL training?

Key findings

• Constitutional AI can produce harmless yet non-evasive assistants without harm labels, using AI feedback to guide RL.
• Critiques and revisions progressively reduce harmfulness, with critiques aiding smaller models more than larger ones.
• AI-generated preference data for harmlessness can match or exceed human-label-based harmlessness performance, especially when using chain-of-thought prompts.
• RL-CAI models achieve higher harmlessness than RLHF and SL-CAI baselines across evaluations, with slight tradeoffs in helpfulness when using CoT.
• Model scaling shows harmlessness and HH scores improve with more revisions, and diversity from multiple principles aids exploration during RL.
• Critiqued revisions generally outperform direct revisions for smaller models; for larger models, gains are similar.