[Paper Review] Learning to Search via Retrospective Imitation

Jialin Song, Ravi Lanka+4 more|arXiv (Cornell University)|Apr 3, 2018

Artificial Intelligence in GamesComputer Science29 references17 citations

TL;DR

This paper proposes retrospective imitation learning, a method that enables reinforcement-free policy improvement by retrospectively refining self-generated search traces after backtracking in combinatorial search spaces. It achieves superior performance and generalization to larger problem instances than imitation learning and commercial solvers like Gurobi, without requiring repeated expert queries.

ABSTRACT

We study the problem of learning a good search policy for combinatorial search spaces. We propose retrospective imitation learning, which, after initial training by an expert, improves itself by learning from extit{retrospective inspections} of its own roll-outs. That is, when the policy eventually reaches a feasible solution in a combinatorial search tree after making mistakes and backtracks, it retrospectively constructs an improved search trace to the solution by removing backtracks, which is then used to further train the policy. A key feature of our approach is that it can iteratively scale up, or transfer, to larger problem sizes than those solved by the initial expert demonstrations, thus dramatically expanding its applicability beyond that of conventional imitation learning. We showcase the effectiveness of our approach on a range of tasks, including synthetic maze solving and combinatorial problems expressed as integer programs.

Motivation & Objective

• To address the challenge of learning effective search policies in combinatorial spaces where reward signals are sparse and expert demonstrations are expensive to obtain.
• To develop a method that improves search policies using only self-generated rollouts and environment feedback, avoiding repeated expert queries.
• To enable scaling of learned policies to larger problem instances than those in the original expert demonstrations.
• To provide theoretical justification for the superiority of retrospective imitation over standard imitation learning under certain conditions.
• To empirically validate the method on A* search, branch-and-bound for integer programs, and minimum vertex cover, showing improved performance and generalization.

Proposed method

• The method uses a retrospective oracle to analyze self-generated rollouts and extract the shortest feasible path to a solution, removing backtracks and incorrect decisions.
• It applies a variant of the DAgger algorithm, where the policy is iteratively refined using these retrospectively constructed optimal traces.
• The approach treats the search process as a sequential decision-making problem, with the policy trained via imitation learning on the refined traces.
• The method enables transfer learning by generating synthetic expert demonstrations from the agent's own rollouts, allowing training on larger problem instances than the original demonstrations.
• It incorporates a selective sampling strategy ('select only') to control search effort and compare fairly with commercial solvers.
• Theoretical analysis establishes that retrospective imitation leads to lower policy error rates than standard imitation learning under specific assumptions.

Experimental results

Research questions

• RQ1 Can a search policy be improved using only its own rollouts and environment feedback, without repeated expert queries?
• RQ2 Can retrospective imitation generalize to problem instances larger than those in the original expert demonstrations?
• RQ3 Does retrospective imitation lead to lower policy error rates and faster convergence than standard imitation learning?
• RQ4 How does the performance of retrospective imitation compare to off-the-shelf solvers like Gurobi and SCIP on large-scale integer programs?
• RQ5 Under what conditions can retrospective imitation provably outperform conventional imitation learning?

Key findings

• Retrospective imitation solved 60% of test instances beyond 12 waypoints in maze tasks, while SMILe failed on 75% of instances beyond 13 waypoints, demonstrating strong extrapolation capability.
• On risk-aware path planning with 14 waypoints (560 binary variables), Retrospective DAgger achieved an optimality gap 50% lower than Gurobi.
• For minimum vertex cover, Retrospective Imitation consistently outperformed conventional imitation learning and commercial solvers, with a 40% performance gap at the largest graph scale.
• The method achieved the lowest number of explored nodes across all problem scales, indicating faster convergence and more efficient search.
• Theoretical analysis confirmed that retrospective imitation leads to lower error rates than standard imitation learning, which correlates with shorter search times.
• Empirical results validated the theoretical claim that lower error rates lead to reduced search effort, as shown by minimal node exploration in maze experiments.