[Paper Review] Regret Circuits: Composability of Regret Minimizers

Gabriele Farina, Christian Kroer+1 more|arXiv (Cornell University)|Nov 6, 2018

Metaheuristic Optimization Algorithms Research32 references4 citations

TL;DR

This paper introduces a calculus for composing regret minimizers over composite convex sets using convexity-preserving operations like intersection, convex hull, and Cartesian product. By treating local regret minimizers as black boxes and combining them via a 'regret circuit' framework, the method enables the construction of globally valid regret minimizers—offering a systematic, modular approach to solving complex optimization problems in game theory and online learning, including generalizations of CFR with strategy constraints.

ABSTRACT

Regret minimization is a powerful tool for solving large-scale problems; it was recently used in breakthrough results for large-scale extensive-form game solving. This was achieved by composing simplex regret minimizers into an overall regret-minimization framework for extensive-form game strategy spaces. In this paper we study the general composability of regret minimizers. We derive a calculus for constructing regret minimizers for composite convex sets that are obtained from convexity-preserving operations on simpler convex sets. We show that local regret minimizers for the simpler sets can be combined with additional regret minimizers into an aggregate regret minimizer for the composite set. As one application, we show that the CFR framework can be constructed easily from our framework. We also show ways to include curtailing (constraining) operations into our framework. For one, they enables the construction of CFR generalization for extensive-form games with general convex strategy constraints that can cut across decision points.

Motivation & Objective

• To develop a general framework for composing regret minimizers over composite convex sets formed by convexity-preserving operations.
• To enable modular construction of regret minimizers by treating local regret minimizers as black boxes with flexible algorithmic choices.
• To provide a systematic method for incorporating strategy constraints that span multiple information sets in extensive-form games.
• To unify and generalize existing regret minimization frameworks such as CFR and Constrained CFR under a single compositional calculus.
• To support both approximate and exact feasibility in constrained optimization via Lagrangian relaxation and projection-based methods.

Proposed method

• The framework uses a calculus of regret minimization based on convexity-preserving operations (e.g., intersection, convex hull, Cartesian product) to build regret minimizers for composite sets from simpler ones.
• It introduces 'regret circuits'—a conceptual and formal representation where local regret minimizers are connected via aggregation operations to form a global regret minimizer.
• The method supports both Lagrangian relaxation and projection-based approaches for handling convex constraints, allowing trade-offs between feasibility and computational cost.
• It enables the use of different regret minimization algorithms (e.g., RM+, Hedge) for different components of the strategy space, enhancing practical performance and theoretical flexibility.
• The framework is formalized in the online convex optimization (OCO) setting, with convergence guarantees derived through compositional reasoning.
• It generalizes the CFR algorithm as a special case, showing that CFR can be derived directly from the calculus by composing simplex regret minimizers with appropriate operations.

Experimental results

Research questions

• RQ1 How can regret minimizers be systematically composed for composite convex sets formed via convexity-preserving operations?
• RQ2 Can the Counterfactual Regret Minimization (CFR) framework be derived as a composition of simpler regret minimizers using a formal calculus?
• RQ3 How can strategy constraints that span multiple information sets in extensive-form games be incorporated into regret minimization frameworks?
• RQ4 What are the trade-offs between Lagrangian relaxation and projection-based methods for enforcing constraints in regret minimization?
• RQ5 Can the framework be used to construct new regret minimizers for equilibrium refinement problems requiring cross-information-set constraints?

Key findings

• The regret calculus enables the construction of a regret minimizer for any composite convex set formed via convexity-preserving operations, using local regret minimizers as black boxes.
• The CFR algorithm can be derived directly from the calculus by composing simplex regret minimizers with appropriate aggregation operations.
• The framework supports both Lagrangian relaxation and projection-based methods for handling constraints, with the latter guaranteeing exact feasibility of all iterates.
• The Lagrangian relaxation variant generalizes the Constrained CFR (CCFR) algorithm and allows for flexible instantiation with any regret minimizer for the multipliers.
• The projection-based method ensures all strategy iterates are feasible, offering a significant advantage over approximate feasibility in CCFR.
• The framework enables the first regret-minimization-based approach to quasi-perfect equilibrium computation with cross-information-set constraints, avoiding heavy-weight operations like matrix inversion.