[Paper Review] Multiple-objective Reinforcement Learning for Inverse Design and Identification

Haoran Wei, Mariefel V. Olarte+1 more|arXiv (Cornell University)|Oct 9, 2019

Computational Drug Discovery Methods20 references4 citations

TL;DR

This paper proposes a curriculum learning-enhanced reinforcement learning framework for multiple-objective inverse molecular design and identification, enabling generation of molecules that satisfy up to 21 constraints (e.g., molecular weight, functional groups) simultaneously. By refining the reward function to prioritize under-learned objectives, the model achieves 83–100% success in identifying unknown molecules, outperforming baseline methods that fail entirely.

ABSTRACT

The aim of the inverse chemical design is to develop new molecules with given optimized molecular properties or objectives. Recently, generative deep learning (DL) networks are considered as the state-of-the-art in inverse chemical design and have achieved early success in generating molecular structures with desired properties in the pharmaceutical and material chemistry fields. However, satisfying a large number (larger than 10 objectives) of molecular objectives is a limitation of current generative models. To improve the model's ability to handle a large number of molecule design objectives, we developed a Reinforcement Learning (RL) based generative framework to optimize chemical molecule generation. Our use of Curriculum Learning (CL) to fine-tune the pre-trained generative network allowed the model to satisfy up to 21 objectives and increase the generative network's robustness. The experiments show that the proposed multiple-objective RL-based generative model can correctly identify unknown molecules with an 83 to 100 percent success rate, compared to the baseline approach of 0 percent. Additionally, this proposed generative model is not limited to just chemistry research challenges; we anticipate that problems that utilize RL with multiple-objectives will benefit from this framework.

Motivation & Objective

• To develop a deep learning framework for inverse molecular design and identification that handles a large number of molecular property constraints.
• To overcome the performance drop in generative models when optimizing for more than 5 objectives simultaneously.
• To improve robustness and convergence in multi-objective molecular generation using reinforcement learning.
• To enable accurate identification of unknown organic molecules from spectral and structural constraints.
• To reduce reliance on human-designed actions by embedding constraints directly into the RL reward function.

Proposed method

• The framework uses a reinforcement learning agent trained on a generative RNN to produce SMILES strings representing molecules.
• Constraints such as molecular weight and functional groups are encoded as components of the RL reward function.
• Curriculum learning is applied by dividing training into phases, gradually increasing the number of active constraints.
• A heuristic is introduced to reshape the constraint function by identifying and prioritizing weakly-learned objectives using a threshold-based selection (ξ = 0.5) on constraint score ratios.
• The refined constraint function, denoted as C′, excludes poorly learned constraints in early phases and reintroduces them later to avoid local optima.
• The model is evaluated using top-5 reward scores and structural similarity to target molecules.

Experimental results

Research questions

• RQ1 Can a reinforcement learning-based generative model effectively optimize for more than 10 molecular objectives simultaneously?
• RQ2 How does curriculum learning improve the performance of multi-objective molecular generation compared to standard RL training?
• RQ3 What impact do the number of curriculum phases and constraint prioritization heuristics have on model convergence and accuracy?
• RQ4 Can the model identify unknown molecules solely from a set of structural and property constraints with high accuracy?
• RQ5 Why do some high-reward predictions fail to achieve high structural similarity, and can this be mitigated?

Key findings

• The proposed model successfully identified 4 out of 5 unknown target molecules with 83–100% success rate, while the baseline achieved 0%.
• The model achieved a maximum top-5 reward score across all five test molecules, indicating effective optimization of multiple objectives.
• The refined curriculum heuristic with adaptive constraint weighting significantly improved structural similarity, especially for targets 4 and 5 where initial performance was poor.
• Models with too many curriculum phases performed worse, suggesting that excessive phase division may trap the agent in suboptimal local optima.
• The use of a threshold-based heuristic (ξ = 0.5) to identify and prioritize weakly-learned constraints improved final performance and robustness.
• The framework demonstrated that RL with curriculum learning can handle up to 21 molecular constraints simultaneously, significantly expanding the scope of current generative models.