[Paper Review] Accelerating Quadratic Optimization with Reinforcement Learning

Jeffrey Ichnowski, Paras Jain+7 more|arXiv (Cornell University)|Jul 22, 2021

Advanced Bandit Algorithms ResearchDecision Sciences42 references20 citations

TL;DR

This paper proposes RLQP, a reinforcement learning-based method that accelerates convergence in quadratic optimization by learning to adapt the ADMM step size parameter (ρ) in real time. Trained on diverse QP problems, RLQP reduces solve time by up to 3x compared to state-of-the-art solvers like OSQP, generalizing effectively across problem classes including QPLIB, Netlib LP, and Maros-Mészáros benchmarks.

ABSTRACT

First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapidly solved. These methods face two persistent challenges: manual hyperparameter tuning and convergence time to high-accuracy solutions. To address these, we explore how Reinforcement Learning (RL) can learn a policy to tune parameters to accelerate convergence. In experiments with well-known QP benchmarks we find that our RL policy, RLQP, significantly outperforms state-of-the-art QP solvers by up to 3x. RLQP generalizes surprisingly well to previously unseen problems with varying dimension and structure from different applications, including the QPLIB, Netlib LP and Maros-Meszaros problems. Code for RLQP is available at https://github.com/berkeleyautomation/rlqp.

Motivation & Objective

• Address the persistent challenge of slow convergence in first-order quadratic optimization methods like OSQP, which require thousands of iterations for high-accuracy solutions.
• Overcome the limitations of manual or heuristic hyperparameter tuning—especially for the ADMM step size ρ—by learning adaptive policies via reinforcement learning.
• Develop a generalizable RL policy that accelerates convergence across diverse QP problem classes without retraining, enabling real-time deployment in control and machine learning applications.
• Investigate the trade-off between generalization and specialization by training policies jointly across problem classes or tailored to specific application domains.
• Demonstrate that learned policies can outperform both non-adaptive and heuristic adaptive solvers in convergence speed and robustness, even on out-of-distribution problems.

Proposed method

• Formulate the QP solving process as a reinforcement learning (RL) environment where the state is the internal solver state (primal and dual variables, constraints), the action is a change to the ADMM step size ρ, and the reward is negative iteration count to minimize solve time.
• Train a neural network policy πθ to map solver states to ρ adaptations using deep RL, with two formulations: scalar (single ρ update per iteration) and vector (per-constraint ρ updates).
• Use a replay buffer and experience replay to stabilize training, with a curriculum learning approach over randomized QP instances to improve sample efficiency.
• Train a general policy on a broad set of QP problems (e.g., QPLIB, Netlib, Maros-Mészáros) for out-of-the-box deployment, and a specialized policy on a target problem class for higher performance in repeated use.
• Integrate the trained policy into the OSQP solver pipeline, replacing fixed or heuristic ρ adaptation with learned, dynamic updates during ADMM iterations.
• Optimize inference speed by using lightweight neural networks and hardware-aware design to ensure the policy overhead does not negate convergence gains.

Experimental results

Research questions

• RQ1 Can reinforcement learning learn effective, generalizable policies to adapt the ADMM step size ρ and accelerate convergence in quadratic optimization?
• RQ2 How does the performance of an RL-based policy compare to non-adaptive and heuristic adaptive solvers across diverse QP problem classes?
• RQ3 To what extent does a single, jointly trained RL policy generalize to previously unseen QP problems with varying dimensions and structures?
• RQ4 Does fine-tuning the policy for a specific problem class yield further performance gains compared to a general policy?
• RQ5 What are the practical limitations of RLQP in terms of training cost, inference latency, and robustness on out-of-distribution problems?

Key findings

• RLQP reduces solve time by up to 3x compared to OSQP across the QPLIB benchmark, with significant improvements on large-scale and ill-conditioned problems.
• The general vector-formulation policy achieves a 1.30× speedup over OSQP on the Netlib LP benchmark, measured by shifted geometric mean of runtimes.
• On the Maros-Mészáros QP benchmark, RLQP’s shifted geometric mean runtime is 1.829× faster than OSQP, demonstrating robustness on poorly scaled problems.
• The general policy generalizes well to out-of-distribution problems, though some timeouts occur due to distributional shift, indicating room for improved generalization.
• Specialized policies trained on specific problem classes further accelerate convergence, suggesting that domain-specific fine-tuning can yield additional gains.
• Despite high training cost (several days on high-end hardware), the resulting policy is efficient at inference, with minimal overhead, making real-time deployment feasible.