[Paper Review] Online 3D Bin Packing with Constrained Deep Reinforcement Learning

Hang Zhao, Qijin She+3 more|arXiv (Cornell University)|Jun 26, 2020

Optimization and Packing Problems51 references13 citations

TL;DR

This paper proposes a constrained deep reinforcement learning (DRL) method for online 3D bin packing under real-time constraints, where items must be packed immediately upon arrival without readjustment. The approach uses a prediction-and-projection scheme to enforce feasibility masks during training, enabling efficient learning of stable, order-dependent packing policies that outperform state-of-the-art methods and human-level performance in preliminary tests.

ABSTRACT

We solve a challenging yet practically useful variant of 3D Bin Packing Problem (3D-BPP). In our problem, the agent has limited information about the items to be packed into the bin, and an item must be packed immediately after its arrival without buffering or readjusting. The item's placement also subjects to the constraints of collision avoidance and physical stability. We formulate this online 3D-BPP as a constrained Markov decision process. To solve the problem, we propose an effective and easy-to-implement constrained deep reinforcement learning (DRL) method under the actor-critic framework. In particular, we introduce a feasibility predictor to predict the feasibility mask for the placement actions and use it to modulate the action probabilities output by the actor during training. Such supervisions and transformations to DRL facilitate the agent to learn feasible policies efficiently. Our method can also be generalized e.g., with the ability to handle lookahead or items with different orientations. We have conducted extensive evaluation showing that the learned policy significantly outperforms the state-of-the-art methods. A user study suggests that our method attains a human-level performance.

Motivation & Objective

• To address the practical challenge of online 3D bin packing where items arrive sequentially and must be packed immediately without buffering or readjustment.
• To incorporate physical stability and order dependence constraints into the packing process, reflecting real-world logistics and robotic applications.
• To develop a deep reinforcement learning framework that learns feasible, stable packing policies under limited lookahead and strict real-time constraints.
• To outperform existing heuristic and learning-based methods in space utilization and packing efficiency on benchmark datasets.
• To evaluate the method’s performance against human intuition through a user study, demonstrating near-human or superior performance.

Proposed method

• The problem is formulated as a constrained Markov decision process (CMDP) to model order dependence and physical stability constraints.
• A prediction-and-projection scheme is introduced: the agent first predicts a feasibility mask for potential placements as an auxiliary task.
• The predicted mask is used to project and modulate the action probabilities from the actor network during training, enforcing only feasible actions.
• The method uses a height-map representation of the bin to encode the 3D packing state and supports multi-bin and lookahead item handling via Monte Carlo Tree Search (MCTS).
• The approach is extended to handle item re-orientation by doubling the action space and learning orientation-specific policies.
• The DRL agent is trained under an on-policy actor-critic framework with constrained policy optimization to ensure stability and feasibility.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn to pack 3D items efficiently in an online setting with limited lookahead and no readjustment?
• RQ2 How effective is the proposed prediction-and-projection scheme in enforcing physical stability and order dependence constraints during training?
• RQ3 Does the method outperform existing heuristic and learning-based approaches in terms of space utilization and number of items packed?
• RQ4 Can the learned policy achieve performance comparable to or exceeding human intuition in real-world packing tasks?
• RQ5 To what extent can the method generalize to multi-bin packing and item re-orientation without architectural changes?

Key findings

• The proposed method achieved 66.3% average space utilization in a real robot test, significantly outperforming the boundary rule baseline (39.2%) and online BPH (43.2%).
• On the RS benchmark, the method achieved 50.5% space utilization and packed 12.2 items on average, surpassing the best baseline (BPH) by 3.5 items and 15.1 percentage points in space utilization.
• On the CUT-1 and CUT-2 benchmarks, the method achieved 73.4% and 66.9% space utilization respectively, outperforming the offline LBP method on CUT-1 and approaching its performance on CUT-2.
• In a user study with 1,851 games, the AI won 1,339 games (72.3% win rate), achieving 68.9% average space utilization, compared to human players’ 52.1%.
• Allowing item re-orientation increased space utilization by 11.6% and the average number of packed items by 3 on the RS dataset, demonstrating the method’s robustness to shape variations.