[Paper Review] Tuning Recurrent Neural Networks with Reinforcement Learning

Natasha Jaques, Shixiang Gu+2 more|arXiv (Cornell University)|Nov 9, 2016

Music and Audio Processing45 citations

TL;DR

This paper proposes a hybrid training approach that combines supervised learning with reinforcement learning (RL) to improve long-term coherence in sequence generation using pre-trained RNNs. By using a pre-trained LSTM for next-note prediction and refining it via RL with music theory-based rewards, the method reduces failure modes and generates more musically coherent melodies while preserving data-learned patterns.

ABSTRACT

The approach of training sequence models using supervised learning and next-step prediction suffers from known failure modes. For example, it is notoriously difficult to ensure multi-step generated sequences have coherent global structure. We propose a novel sequence-learning approach in which we use a pre-trained Recurrent Neural Network (RNN) to supply part of the reward value in a Reinforcement Learning (RL) model. Thus, we can refine a sequence predictor by optimizing for some imposed reward functions, while maintaining good predictive properties learned from data. We propose efficient ways to solve this by augmenting deep Q-learning with a cross-entropy reward and deriving novel off-policy methods for RNNs from KL control. We explore the usefulness of our approach in the context of music generation. An LSTM is trained on a large corpus of songs to predict the next note in a musical sequence. This Note RNN is then refined using our method and rules of music theory. We show that by combining maximum likelihood (ML) and RL in this way, we can not only produce more pleasing melodies, but significantly reduce unwanted behaviors and failure modes of the RNN, while maintaining information learned from data.

Motivation & Objective

• To address the limitation of RNNs in generating coherent long-term sequences when trained solely via next-step prediction.
• To maintain the predictive accuracy learned from large-scale data while improving global structure in generated sequences.
• To integrate domain-specific knowledge (e.g., music theory rules) into sequence modeling via reinforcement learning.
• To develop efficient RL methods tailored for RNNs that support off-policy training and KL control.
• To evaluate the approach in music generation, where coherence and structural quality are critical.

Proposed method

• A pre-trained LSTM is used as the policy network to predict the next note in a musical sequence, initialized with maximum likelihood (ML) training on a large corpus of songs.
• Reinforcement learning is applied to refine the RNN policy by optimizing for a reward function based on music theory rules.
• The reward function incorporates a cross-entropy component to align with the pre-trained model’s predictive behavior, preserving data-learned patterns.
• Novel off-policy RL methods are derived from KL control principles to improve sample efficiency and training stability in RNNs.
• The method enables joint optimization of likelihood and reward, balancing data fidelity with structural quality.
• The approach is implemented using deep Q-learning with modifications to handle sequential decision-making and RNN dynamics.

Experimental results

Research questions

• RQ1 Can combining maximum likelihood pre-training with reinforcement learning improve long-term coherence in RNN-generated sequences?
• RQ2 How can domain-specific knowledge (e.g., music theory) be effectively integrated into sequence modeling to guide generation?
• RQ3 What are the effects of using a pre-trained RNN as a reward component in RL fine-tuning?
• RQ4 Can off-policy RL methods be adapted effectively for RNNs to improve training efficiency and stability?
• RQ5 To what extent does the hybrid ML-RL approach reduce failure modes in sequence generation while preserving data-learned representations?

Key findings

• The hybrid ML-RL approach significantly improves the musical quality and coherence of generated melodies compared to standard next-step prediction.
• The method reduces common failure modes in RNNs, such as repetitive or incoherent note patterns, by enforcing structural constraints via reward shaping.
• By preserving the pre-trained RNN’s predictive behavior through a cross-entropy reward, the model maintains strong data-fitting properties.
• The use of off-policy RL methods enables stable and efficient training, even with sequential data and complex RNN dynamics.
• The integration of music theory rules into the reward function leads to more harmonically and rhythmically plausible sequences.
• The approach achieves a balance between data fidelity and structural quality, outperforming both pure ML and pure RL baselines in qualitative and quantitative evaluations.