[Paper Review] Toolpath design for additive manufacturing using deep reinforcement learning

Mojtaba Mozaffar, Ablodghani Ebrahimi+1 more|arXiv (Cornell University)|Sep 30, 2020

Additive Manufacturing and 3D Printing Technologies10 references4 citations

TL;DR

This paper proposes a deep reinforcement learning framework for optimizing toolpath design in metal-based additive manufacturing by modeling the process as a reinforcement learning problem. Using DQN, SAC, and PPO algorithms, it demonstrates that model-free RL achieves high performance, especially with dense reward structures, where DQN and PPO surpass manual zig-zag toolpaths, while SAC underperforms due to instability in this task context.

ABSTRACT

Toolpath optimization of metal-based additive manufacturing processes is currently hampered by the high-dimensionality of its design space. In this work, a reinforcement learning platform is proposed that dynamically learns toolpath strategies to build an arbitrary part. To this end, three prominent model-free reinforcement learning formulations are investigated to design additive manufacturing toolpaths and demonstrated for two cases of dense and sparse reward structures. The results indicate that this learning-based toolpath design approach achieves high scores, especially when a dense reward structure is present.

Motivation & Objective

• To address the high-dimensional design space challenge in metal-based additive manufacturing toolpath optimization.
• To investigate whether model-free reinforcement learning can effectively learn optimal toolpath strategies without prior knowledge of the environment.
• To compare the performance of three state-of-the-art deep RL algorithms—DQN, SAC, and PPO—on toolpath generation under both dense and sparse reward settings.
• To evaluate sample efficiency and convergence behavior of these algorithms in a simulated AM environment.
• To demonstrate that engineered dense rewards significantly improve learning performance in complex manufacturing tasks.

Proposed method

• The toolpath design problem is formulated as a Markov Decision Process (MDP), where the agent learns to select actions (toolpath moves) based on environmental states.
• A custom simulation environment is developed to model the additive manufacturing process, including state representation based on the build’s current geometry and deposition history.
• Three model-free deep RL algorithms—Deep Q-Network (DQN), Soft Actor-Critic (SAC), and Proximal Policy Optimization (PPO)—are implemented and trained using the Tensorflow library.
• Dense and sparse reward functions are designed: dense rewards provide feedback at each step based on geometric and process quality metrics, while sparse rewards only signal success or failure at the end.
• Hyperparameters are tuned to maximize cumulative reward, and SAC uses entropy regularization via a temperature parameter to balance exploration and exploitation.
• The agent’s policy is updated using off-policy (DQN, SAC) or on-policy (PPO) learning, with experience replay and target networks used in DQN and SAC for stability.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively optimize toolpath strategies in high-dimensional additive manufacturing design spaces?
• RQ2 How do different model-free RL algorithms—DQN, SAC, and PPO—perform in learning optimal toolpaths under dense versus sparse reward structures?
• RQ3 Does the presence of dense, step-by-step rewards significantly improve learning efficiency and final performance in toolpath design?
• RQ4 Why does SAC, despite its strong performance in other RL benchmarks, underperform in this specific toolpath optimization task?
• RQ5 To what extent can on-policy algorithms like PPO achieve better solutions despite requiring significantly more training samples?

Key findings

• In the dense reward setting, DQN and PPO outperform a manually coded zig-zag toolpath, achieving scores of 59.31 and 62.98, respectively.
• DQN reaches a stable solution using 10 times fewer samples than PPO, indicating superior sample efficiency in this task.
• SAC achieves the lowest score (38.71) in the dense setting, suggesting it is ill-suited for this specific toolpath optimization problem.
• In the sparse reward setting, PPO achieves the highest score (40.54), while SAC performs worst (5.11), indicating poor learning capability under sparse feedback.
• The results show that dense rewards are critical for effective learning, and model-free RL is feasible for high-dimensional manufacturing design when reward shaping is applied.
• PPO’s high sample requirement limits its practicality to simulation-based environments, highlighting the need for hybrid model-free/model-based approaches with real experimental data.