[Paper Review] Deep Reinforcement Learning for Process Synthesis

Laurence Illing Midgley|arXiv (Cornell University)|Sep 23, 2020

Process Optimization and Integration8 references4 citations

TL;DR

This paper introduces Distillation Gym, a reinforcement learning environment that enables deep RL agents to design distillation trains for multicomponent separations by interfacing with process simulators (COCO and ChemSep). Using Soft Actor-Critic (SAC), the agent successfully learns to generate high-recovery, profitable designs for benzene-toluene-p-xylene and hydrocarbon separation problems, demonstrating the feasibility of framing process synthesis as an RL task and proposing Chemical Engineering Gym as a general-purpose toolkit for broader application.

ABSTRACT

This paper demonstrates the application of reinforcement learning (RL) to process synthesis by presenting Distillation Gym, a set of RL environments in which an RL agent is tasked with designing a distillation train, given a user defined multi-component feed stream. Distillation Gym interfaces with a process simulator (COCO and ChemSep) to simulate the environment. A demonstration of two distillation problem examples are discussed in this paper (a Benzene, Toluene, P-xylene separation problem and a hydrocarbon separation problem), in which a deep RL agent is successfully able to learn within Distillation Gym to produce reasonable designs. Finally, this paper proposes the creation of Chemical Engineering Gym, an all-purpose reinforcement learning software toolkit for chemical engineering process synthesis.

Motivation & Objective

• To demonstrate that reinforcement learning can be effectively applied to process synthesis by formulating distillation train design as a sequential decision-making task.
• To develop a functional RL environment (Distillation Gym) that interfaces with process simulators (COCO and ChemSep) to enable simulation-based training.
• To validate the approach on real-world separation problems—benzene-toluene-p-xylene and hydrocarbon mixtures—showcasing the agent’s ability to learn profitable, high-recovery designs.
• To propose Chemical Engineering Gym as a general-purpose RL toolkit for chemical engineering process synthesis, expanding beyond distillation to include diverse unit operations and process topologies.
• To advocate for a shift from traditional optimization framing (parameter tuning) to decision-sequence framing (process synthesis as action sequences), enabling more open-ended and innovative design exploration.

Proposed method

• The environment is structured as a Markov Decision Process (MDP), where the agent observes the state (feed composition, conditions), selects actions (column configuration, splits), and receives rewards based on product purity, recovery, and revenue.
• The agent employs Soft Actor-Critic (SAC), a deep RL algorithm that uses an actor-critic architecture with entropy regularization to balance exploration and exploitation, improving sample efficiency and stability.
• The critic network estimates the soft Q-value using a double Q-network setup with target networks and soft updates to stabilize training, minimizing the Bellman error over a replay buffer of experiences.
• The actor network uses a reparameterization trick to enable differentiable policy optimization, allowing gradient-based updates to the policy network using noise-embedded actions.
• The environment integrates with COCO and ChemSep simulators to compute process performance (separation, energy, cost) after each action, providing accurate state transitions and reward signals.
• Problem specification is user-defined, including component properties, feed stream conditions, product purity targets, and market prices, enabling flexible and generalizable problem instantiation.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively solve process synthesis problems by formulating distillation train design as a sequential decision-making task?
• RQ2 To what extent can an RL agent learn to generate high-recovery, profitable distillation configurations without prior optimization or heuristic rules?
• RQ3 How does the integration of a process simulator (COCO/ChemSep) with an RL agent enable accurate and realistic performance evaluation during training?
• RQ4 What are the key design trade-offs (e.g., recovery vs. cost) that an RL agent can learn to balance in complex multicomponent separations?
• RQ5 Can the RL-based approach be generalized beyond distillation to other unit operations and process topologies, and what infrastructure is needed to support such scalability?

Key findings

• The deep RL agent successfully learned to design a distillation train for the benzene-toluene-p-xylene (BTX) separation problem, achieving 98.2% recovery for benzene, 99.7% for toluene, and over 99.9% for p-xylene, with a total revenue of $0.45 million.
• For the hydrocarbon separation problem, the agent achieved high recoveries: 98.9% for propane, 97.3% for isobutane, 91.1% for n-butane, 99.6% for isopentane, and 97.0% for n-pentane, generating total revenue of $1.588 billion.
• The agent’s best design for the BTX problem was autogenerate as a single-block flowsheet with a valve-distillation column pair, demonstrating structural simplicity and feasibility.
• The agent’s performance was robust across multiple training runs, indicating stable learning dynamics in the Distillation Gym environment with SAC.
• The results validate that RL can discover non-trivial, profitable process configurations without relying on predefined heuristics or MINLP-based optimization.
• The authors conclude that framing process synthesis as a sequence of decisions, rather than parameter optimization, enables more flexible and innovative design space exploration, especially when extended to a broader toolkit like Chemical Engineering Gym.