[Paper Review] Quinoa: a Q-function You Infer Normalized Over Actions

Jonas Degrave, Abbas Abdolmaleki+3 more|arXiv (Cornell University)|Nov 5, 2019

Control Systems and Identification15 references4 citations

TL;DR

Quinoa proposes a novel actor-critic reinforcement learning algorithm that learns a soft Q-function with a normalizing flow policy, enabling closed-form optimal policy inference without separate policy optimization. By using normalizing flows to model complex, multimodal action distributions, Quinoa simplifies training and achieves performance comparable to state-of-the-art methods while enabling rich, non-Gaussian exploration in continuous control tasks.

ABSTRACT

We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.

Motivation & Objective

• To eliminate the need for separate policy optimization steps in off-policy actor-critic algorithms.
• To enable exact, fast sampling from complex, multimodal policies in continuous action spaces.
• To develop a unified learning objective that jointly optimizes value and policy via temporal difference learning.
• To support arbitrary, expressive policy distributions using normalizing flows while maintaining closed-form optimal policy inference.
• To demonstrate improved policy expressivity and training stability in continuous control benchmarks.

Proposed method

• The method uses a relative entropy regularized RL objective to define a soft Q-function that supports closed-form optimal policy derivation.
• It parametrizes the policy using normalizing flows (specifically Real NVP) conditioned on states, enabling universal density approximation.
• The soft Q-function is learned via temporal difference learning by minimizing the squared TD error over replay buffer transitions.
• The optimal policy is obtained by normalizing the soft Q-function over actions, eliminating the need for iterative policy optimization.
• The value function and policy are trained jointly using a target network and gradient clipping to stabilize training.
• The approach enables exact, efficient sampling from the policy while supporting non-Gaussian, multimodal, and skewed action distributions.

Experimental results

Research questions

• RQ1 Can a soft Q-function be learned such that the optimal policy is obtainable in closed form without iterative optimization?
• RQ2 Can normalizing flows enable expressive, multimodal policies in continuous control while maintaining efficient sampling and training?
• RQ3 Does eliminating the policy optimization step improve training stability and performance in continuous control benchmarks?
• RQ4 Can the proposed method achieve performance comparable to state-of-the-art algorithms like SVG(0) while supporting richer policy distributions?
• RQ5 How do the learned policies differ in structure (e.g., multimodal, non-Gaussian) from those learned with standard Gaussian policies?

Key findings

• Quinoa achieves performance comparable to SVG(0) on the Cheetah and Walker tasks in the DeepMind Control Suite, with slightly lower performance on the Hopper task.
• The policy learned by Quinoa exhibits non-Gaussian behavior, including high skewness, non-linearly correlated noise, and finite support in action space.
• The policy displays multimodal behavior in certain states, such as the Walker environment, where it explores actions in multiple corners of the action space.
• Some action dimensions collapse during training while others remain high-variance to preserve entropy and exploration.
• The method enables exact and fast sampling from the policy due to the invertibility of normalizing flows, even with complex distributions.
• The closed-form policy inference removes the need for separate policy optimization, simplifying the training pipeline.