[Paper Review] An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning

Yihua Zhang, Prashant Khanduri+4 more|arXiv (Cornell University)|Aug 1, 2023

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper provides a comprehensive introduction to bi-level optimization (BLO) for signal processing and machine learning, covering foundational theory, tractable problem classes, optimization algorithms, and practical applications. It demonstrates how BLO models hierarchical decision-making in tasks like adversarial robustness, model generalization, and coreset selection, offering theoretical insights and identifying key open challenges for future research.

ABSTRACT

Recently, bi-level optimization (BLO) has taken center stage in some very exciting developments in the area of signal processing (SP) and machine learning (ML). Roughly speaking, BLO is a classical optimization problem that involves two levels of hierarchy (i.e., upper and lower levels), wherein obtaining the solution to the upper-level problem requires solving the lower-level one. BLO has become popular largely because it is powerful in modeling problems in SP and ML, among others, that involve optimizing nested objective functions. Prominent applications of BLO range from resource allocation for wireless systems to adversarial machine learning. In this work, we focus on a class of tractable BLO problems that often appear in SP and ML applications. We provide an overview of some basic concepts of this class of BLO problems, such as their optimality conditions, standard algorithms (including their optimization principles and practical implementations), as well as how they can be leveraged to obtain state-of-the-art results for a number of key SP and ML applications. Further, we discuss some recent advances in BLO theory, its implications for applications, and point out some limitations of the state-of-the-art that require significant future research efforts. Overall, we hope that this article can serve to accelerate the adoption of BLO as a generic tool to model, analyze, and innovate on a wide array of emerging SP and ML applications.

Motivation & Objective

• To provide a unified overview of bi-level optimization (BLO) as a foundational framework for modeling hierarchical decision-making in signal processing and machine learning.
• To present key theoretical concepts, including optimality conditions and convergence guarantees, for a tractable class of BLO problems with convex or non-convex components.
• To survey state-of-the-art BLO algorithms and their practical implementations in real-world SP and ML applications such as adversarial training, model generalization, and resource allocation.
• To identify critical open problems in BLO, including handling non-linear constraints, non-singleton lower-level solutions, multi-level hierarchies, and decentralized/federated settings.
• To accelerate the adoption of BLO as a standard tool for modeling, analyzing, and innovating in emerging SP and ML applications.

Proposed method

• Formalizes BLO as a two-level optimization problem: minimize an upper-level objective that depends on the solution of a lower-level optimization subproblem.
• Introduces the canonical BLO formulation with upper-level variables θ and lower-level variables φ, where φ* is the optimal solution to the lower-level problem constrained by h(θ,φ) ≤ 0.
• Reviews standard BLO algorithms based on implicit differentiation and nested optimization, emphasizing their optimization principles and practical implementation strategies.
• Applies BLO to real-world problems such as coreset selection, where the upper level selects data weights and the lower level trains a model on the selected subset.
• Analyzes the theoretical properties of BLO in non-convex and black-box settings, and discusses convergence under realistic assumptions.
• Proposes future algorithmic directions, including decentralized and federated BLO frameworks, and extensions to multi-level and overparameterized problems.

Figure 1: An taxonomy of the solvers and application tasks for BLO.

Experimental results

Research questions

• RQ1 How can bi-level optimization effectively model hierarchical decision-making in signal processing and machine learning applications?
• RQ2 What are the key theoretical challenges in solving BLO problems with non-linear constraints, non-singleton lower-level solutions, or black-box components?
• RQ3 How can BLO be leveraged to improve model generalization, adversarial robustness, and data efficiency in deep learning?
• RQ4 What are the limitations of current BLO algorithms in large-scale and distributed settings, and how can they be addressed?
• RQ5 In what ways can BLO be extended to multi-level optimization problems such as transfer learning or mixture-of-experts training?

Key findings

• Bi-level optimization enables effective modeling of hierarchical structures in SP and ML, such as in coreset selection, where data selection (upper level) depends on model training (lower level).
• The paper identifies non-linear and coupled constraints as major challenges in BLO, with current methods primarily focused on linear constraints and limited theoretical guarantees.
• Non-singleton lower-level solutions (NS-BLO) remain an open problem, as existing algorithms lack theoretical grounding and scalability, representing a key research gap.
• Extending BLO to multi-level problems—such as dataset pruning in transfer learning—introduces new complexity and requires novel algorithmic frameworks.
• Large-scale and distributed data settings demand decentralized and federated BLO algorithms, which remain underdeveloped and represent a critical frontier for future research.
• Theoretical analysis of BLO in overparameterized settings, such as those exhibiting double descent, is an emerging and promising direction for future study.