[Paper Review] Modeling Friends and Foes

Pedro A. Ortega, Shane Legg|arXiv (Cornell University)|Jun 30, 2018

Advanced Bandit Algorithms Research31 references4 citations

TL;DR

This paper introduces a continuous, information-theoretic framework to model environmental attitudes—ranging from fully friendly to fully adversarial—by treating agent-environment interaction as a one-shot game with information constraints. It derives optimal equilibrium strategies for both agent and environment using bounded rationality and Lagrangian optimization, showing that non-trivial, behaviorally distinct strategies emerge under different attitudes, even in simple bandit settings.

ABSTRACT

How can one detect friendly and adversarial behavior from raw data? Detecting whether an environment is a friend, a foe, or anything in between, remains a poorly understood yet desirable ability for safe and robust agents. This paper proposes a definition of these environmental "attitudes" based on an characterization of the environment's ability to react to the agent's private strategy. We define an objective function for a one-shot game that allows deriving the environment's probability distribution under friendly and adversarial assumptions alongside the agent's optimal strategy. Furthermore, we present an algorithm to compute these equilibrium strategies, and show experimentally that both friendly and adversarial environments possess non-trivial optimal strategies.

Motivation & Objective

• To formalize the concept of environmental attitudes—friendly, adversarial, or indifferent—as a continuous spectrum based on how environments react to an agent’s private strategy.
• To develop a game-theoretic model where both agent and environment act rationally under bounded rationality, optimizing under information constraints.
• To derive equilibrium strategies for both agent and environment using variational inference and Lagrangian optimization, enabling computation of optimal behavior under different environmental attitudes.
• To demonstrate empirically that these strategies exhibit qualitatively distinct behaviors depending on the environment’s attitude and information constraints.
• To provide a foundation for AI safety by enabling agents to detect and respond to reactive environments, improving robustness in uncertain or adversarial settings.

Proposed method

• Models the agent-environment interaction as a one-shot game with private strategies, where the environment’s response depends on the agent’s policy through a hidden parameter.
• Introduces a joint objective function that incorporates utility, prior beliefs (Q(x), Q(z)), and bounded rationality via inverse temperature β, using the principle of maximum causal entropy.
• Derives best-response functions for both agent and environment via Lagrangian relaxation of normalization constraints, yielding closed-form expressions involving exponentials of expected utility.
• Applies Brouwer’s fixed-point theorem to prove existence of a unique equilibrium in the strategy space, ensuring stable solutions.
• Uses iterative fixed-point algorithms to compute equilibrium strategies, with convergence guaranteed by continuity and compactness of the strategy space.
• Empirically evaluates the model on two-armed bandit environments, varying the environment’s reactivity and observing strategy shifts in response to predictability.

Experimental results

Research questions

• RQ1 How can we define and quantify environmental attitudes such as friendly, adversarial, or indifferent in a continuous, mathematically rigorous way?
• RQ2 What are the optimal strategies for an agent and environment when their interaction is constrained by bounded rationality and information asymmetry?
• RQ3 How do equilibrium strategies change as the environment shifts from friendly to adversarial, and what behavioral patterns emerge?
• RQ4 Can we detect whether an environment is reactive (i.e., responds to the agent’s strategy) using observable data, and how can this be formalized?
• RQ5 What is the impact of information constraints on the optimality of agent strategies, and can suboptimal performance be avoided by modeling environmental attitudes correctly?

Key findings

• The model successfully derives equilibrium strategies for both agent and environment using variational inference and Lagrangian optimization, with closed-form solutions under bounded rationality.
• The equilibrium strategies exhibit non-trivial behavior: agents adapt their policies based on the environment’s reactivity, with distinct patterns emerging under friendly versus adversarial assumptions.
• In a two-armed bandit experiment, deterministic strategies were exploited more severely by adversarial environments (e.g., bandit D), while friendly environments (e.g., bandit B) improved rewards under predictable behavior.
• The degree of adversarial reaction correlates with the mutual information between the agent’s strategy and the environment’s response, with higher information flow indicating stronger reactivity.
• Even in the friendly case, failing to model the environment’s attitude leads to strictly suboptimal strategies, highlighting the importance of fine-grained attitude discrimination.
• The framework supports a continuous spectrum of attitudes via a single real-valued parameter (β), enabling smooth interpolation between full friendliness and full adversariality.