[Paper Review] Neuro-algorithmic Policies enable Fast Combinatorial Generalization

Marin Vlastelica, Michal Rolínek+1 more|arXiv (Cornell University)|Feb 15, 2021

Machine Learning and Algorithms3 references4 citations

TL;DR

This paper proposes Neuro-Algorithmic Policies (NAP), a hybrid architecture combining a deep neural network with a differentiable time-dependent shortest path (TDSP) solver to enable fast combinatorial generalization in reinforcement learning. By end-to-end training via blackbox differentiation, NAP achieves strong zero-shot generalization to unseen environment variations with orders-of-magnitude fewer expert demonstrations than standard imitation learning baselines.

ABSTRACT

Although model-based and model-free approaches to learning the control of systems have achieved impressive results on standard benchmarks, generalization to task variations is still lacking. Recent results suggest that generalization for standard architectures improves only after obtaining exhaustive amounts of data. We give evidence that generalization capabilities are in many cases bottlenecked by the inability to generalize on the combinatorial aspects of the problem. Furthermore, we show that for a certain subclass of the MDP framework, this can be alleviated by neuro-algorithmic architectures. Many control problems require long-term planning that is hard to solve generically with neural networks alone. We introduce a neuro-algorithmic policy architecture consisting of a neural network and an embedded time-dependent shortest path solver. These policies can be trained end-to-end by blackbox differentiation. We show that this type of architecture generalizes well to unseen variations in the environment already after seeing a few examples.

Motivation & Objective

• To address the poor generalization of standard deep RL and imitation learning methods when faced with unseen environmental variations.
• To investigate whether embedding combinatorial algorithms like shortest path solvers into neural policies can improve generalization in low-data regimes.
• To demonstrate that neuro-algorithmic architectures can achieve strong generalization by shifting combinatorial complexity to efficient algorithms.
• To evaluate the effectiveness of differentiable planners in dynamic, image-based control environments.

Proposed method

• The method uses a deep neural network to generate a time-evolving graph from input observations, where edge weights represent time-dependent costs.
• A differentiable time-dependent shortest path (TDSP) solver computes optimal trajectories on the latent graph, with gradients backpropagated via blackbox differentiation.
• The entire architecture is trained end-to-end using imitation learning from a small number of expert trajectories.
• The latent graph structure assumes a known topological grid with fixed actions, and the solver operates on evolving costs derived from the network.
• The approach leverages the combinatorial inductive bias of shortest path algorithms to improve generalization without requiring explicit model learning.
• The method is applied to both static and dynamic environments, including image-based game environments.

Experimental results

Research questions

• RQ1 Can embedding a differentiable shortest path solver into a neural policy improve generalization to unseen environment variations?
• RQ2 How does the performance of neuro-algorithmic policies compare to standard imitation learning baselines in low-data regimes?
• RQ3 Does the use of a combinatorial planner lead to more optimal (shorter) trajectories in generalization settings?
• RQ4 How sensitive is the method to the planning horizon length in dynamic environments?
• RQ5 Can neuro-algorithmic policies generalize effectively in image-based control tasks with minimal expert demonstrations?

Key findings

• NAP achieved near 80% success rate in the Maze environment with only 100 training levels, a performance level never reached by PPO or DrAC even after 1,000 levels.
• NAP trained on 1,000 levels matched the performance of PPO trained on 200,000 levels in the Maze environment.
• In the Crash Jewel Hunt 5×5 environment, NAP solved 33% of 1,000 test levels with just 30 expert trajectories, while the behavior cloning baseline solved fewer than 5%.
• NAP generalizes with significantly shorter path lengths than PPO and DrAC∗, even on solved levels, indicating better path optimality.
• Longer planning horizons improved performance in dynamic environments, but provided no benefit in static environments like Maze, as expected.
• The method shows strong generalization even with minimal data, demonstrating that combinatorial inductive biases from planners are key to low-data generalization.