[Paper Review] Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models

Gabriel Lima Guimaraes, Benjamín Sánchez-Lengeling+3 more|arXiv (Cornell University)|May 30, 2017

Topic Modeling29 references495 citations

TL;DR

ORGAN extends SeqGAN by combining adversarial rewards with domain-specific objective rewards in a reinforcement learning framework to steer sequence generation (molecules and music) toward desired properties while maintaining data-likeness and diversity, using Wasserstein GANs for stability.

ABSTRACT

In unsupervised data generation tasks, besides the generation of a sample based on previous observations, one would often like to give hints to the model in order to bias the generation towards desirable metrics. We propose a method that combines Generative Adversarial Networks (GANs) and reinforcement learning (RL) in order to accomplish exactly that. While RL biases the data generation process towards arbitrary metrics, the GAN component of the reward function ensures that the model still remembers information learned from data. We build upon previous results that incorporated GANs and RL in order to generate sequence data and test this model in several settings for the generation of molecules encoded as text sequences (SMILES) and in the context of music generation, showing for each case that we can effectively bias the generation process towards desired metrics.

Motivation & Objective

• Motivate the need to guide sequence generators toward domain-specific metrics without losing resemblance to the data distribution.
• Propose ORGAN to blend discriminator-based rewards with explicit objective rewards in a reinforcement learning setting.
• Demonstrate improvements in target properties and diversity for discrete sequence generation (molecules and music).
• Investigate training stability via Wasserstein distance and diversity-preserving penalties.

Proposed method

• Build on SeqGAN by modeling the generator as an RL policy trained with rewards.
• Define the combined reward R(Y1:T) = λ · Dφ(Y1:T) + (1 − λ) · Oi(Y1:T).
• Use Monte Carlo rollouts to estimate Q for partial sequences and guide policy gradient updates.
• Promote diversity by penalizing non-unique sequences (repeated samples).
• Adopt Wasserstein-1 distance (WGAN) for discriminator training to stabilize GAN dynamics.
• Implementation uses an LSTM-based generator (Gθ) and a CNN-based discriminator (Dφ) with standard optimization (Adam).

Experimental results

Research questions

• RQ1 Can ORGAN steer sequence generation toward domain-specific objectives without excessively deviating from the original data distribution?
• RQ2 Does combining discriminator rewards with objective rewards improve both targeted metrics and sample diversity compared to baselines?
• RQ3 What impact does using the Wasserstein distance have on training stability and sample quality in ORGAN?
• RQ4 How does ORGAN perform in different domains (molecular SMILES strings and musical melodies) with varying objective signals?
• RQ5 Is there an optimal balance parameter λ that jointly maximizes target objectives and data-likelihood?

Key findings

• ORGAN improves targeted domain metrics relative to MLE and SeqGAN across molecular and musical tasks.
• WGAN-ORGAN variants show enhanced diversity and often better property distributions than non-WGAN baselines.
• Naive RL can overfit to simple patterns, while ORGAN preserves diversity via adversarial guidance and non-unique sequence penalties.
• Alternating multi-objective training (rotating objectives) yields high gains across metrics, approaching best single-objective models.
• Tuning λ trades off objective optimization and data-likelihood, with an optimal value depending on dataset and metric.