[Paper Review] Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games

Ninell Oldenburg, Tan Zhi‐Xuan|arXiv (Cornell University)|Feb 20, 2024

Bayesian Modeling and Causal Inference4 citations

TL;DR

This paper proposes a Bayesian rule induction framework for multi-agent reinforcement learning in Markov games, enabling agents to learn and sustain shared normative systems through approximate inference over obligative and prohibitive norms. By assuming shared normativity and updating beliefs via observed compliance/violation patterns, agents achieve rapid, sample-efficient norm learning and stable cooperation, outperforming model-free baselines in long-horizon environments.

ABSTRACT

A universal feature of human societies is the adoption of systems of rules and norms in the service of cooperative ends. How can we build learning agents that do the same, so that they may flexibly cooperate with the human institutions they are embedded in? We hypothesize that agents can achieve this by assuming there exists a shared set of norms that most others comply with while pursuing their individual desires, even if they do not know the exact content of those norms. By assuming shared norms, a newly introduced agent can infer the norms of an existing population from observations of compliance and violation. Furthermore, groups of agents can converge to a shared set of norms, even if they initially diverge in their beliefs about what the norms are. This in turn enables the stability of the normative system: since agents can bootstrap common knowledge of the norms, this leads the norms to be widely adhered to, enabling new entrants to rapidly learn those norms. We formalize this framework in the context of Markov games and demonstrate its operation in a multi-agent environment via approximately Bayesian rule induction of obligative and prohibitive norms. Using our approach, agents are able to rapidly learn and sustain a variety of cooperative institutions, including resource management norms and compensation for pro-social labor, promoting collective welfare while still allowing agents to act in their own interests.

Motivation & Objective

• To address the challenge of enabling autonomous agents to learn and comply with decentralized, shared social norms in multi-agent environments.
• To model norm learning as a rational, Bayesian inference process that infers structured rules from observed behavior, rather than relying on reactive or model-free learning.
• To demonstrate that agents can bootstrap common knowledge of norms through shared belief formation, enabling rapid onboarding of new agents.
• To formalize normative systems as rule-based structures (obligations and prohibitions) integrated into Markov games for coordinated, goal-directed behavior.
• To evaluate the framework in long-horizon multi-agent environments, showing sample efficiency and stability of normative cooperation.

Proposed method

• Formalize norm-augmented Markov games by extending standard MDPs with normative constraints, including prohibitive and obligative rules.
• Model agents as performing norm-compliant planning by switching between reward-maximizing and norm-satisfying modes based on current normative beliefs.
• Implement approximate Bayesian inference to update posterior probabilities over candidate norms using observations of others’ actions.
• Use a probabilistic rule induction mechanism that treats normative content as hypotheses, with likelihoods based on observed compliance or violation patterns.
• Integrate normative beliefs into policy learning, allowing agents to balance self-interest with normative compliance through structured, interpretable rules.
• Leverage symbolic rule representations to enable generalization, communication, and interpretability of norms across agents.

(a) Norm-Augmented Markov Game

Experimental results

Research questions

• RQ1 Can agents learn shared social norms through Bayesian inference of rule structures from observational data in multi-agent environments?
• RQ2 How does assuming shared normativity accelerate norm learning and improve sample efficiency compared to model-free approaches?
• RQ3 To what extent can agents converge on a common normative system even when initially holding divergent beliefs about the rules?
• RQ4 Can norm-compliant behavior emerge and be sustained in long-horizon multi-agent settings without centralized enforcement?
• RQ5 How do symbolic, rule-based norms compare to habit-based or reactive learning in terms of interpretability, generalization, and stability?

Key findings

• Agents using approximate Bayesian rule induction learned and sustained cooperative norms—such as resource management and compensation for pro-social labor—significantly faster than model-free baselines.
• The approach achieved orders of magnitude greater sample efficiency than model-free norm learning methods, requiring far less experience to converge.
• Shared normativity enabled the emergence of common knowledge of norms: once a critical mass of agents inferred the same norms, compliance stabilized and new agents could rapidly learn them.
• The framework successfully supported norm-compliant planning by switching between reward-oriented and obligation-oriented modes, ensuring both self-interest and norm compliance.
• Symbolic rule representations allowed for interpretability, generalization, and communication of norms, supporting richer normative cognition than purely reactive learning.
• Empirical evaluation in DeepMind's Melting Pot simulator confirmed the feasibility and robustness of the approach in complex, long-horizon multi-agent environments.

(b) Social Norms