[Paper Review] Hierarchical Planning in the IPC

Daniel Höller, Gregor Behnke+5 more|arXiv (Cornell University)|Sep 10, 2019

AI-based Problem Solving and Planning24 references4 citations

TL;DR

This paper proposes a new HTN (Hierarchical Task Network) planning track for the International Planning Competition (IPC) to standardize input languages, benchmark domains, and evaluation metrics. By introducing a unified competition framework—including community-sourced domains and a 'Bring Your Own Benchmark' model—it aims to unify the HTN planning community, enable fair comparisons, and drive future research through standardized, verifiable, and expressive problem sets.

ABSTRACT

Over the last year, the amount of research in hierarchical planning has increased, leading to significant improvements in the performance of planners. However, the research is diverging and planners are somewhat hard to compare against each other. This is mostly caused by the fact that there is no standard set of benchmark domains, nor even a common description language for hierarchical planning problems. As a consequence, the available planners support a widely varying set of features and (almost) none of them can solve (or even parse) any problem developed for another planner. With this paper, we propose to create a new track for the IPC in which hierarchical planners will compete. This competition will result in a standardised description language, broader support for core features of that language among planners, a set of benchmark problems, a means to fairly and objectively compare HTN planners, and for new challenges for planners.

Motivation & Objective

• To address the lack of standardization in hierarchical planning research, where planners use incompatible input languages and cannot solve each other's problems.
• To unify the HTN planning community by establishing a common input language and core features supported by all competing planners.
• To create a standardized benchmark suite of domains, including both community-provided and 'Bring Your Own Benchmark' (BYOB) domains, to enable fair and objective planner comparisons.
• To foster long-term progress in HTN planning by identifying weaknesses in current approaches through competition, guiding future research directions.
• To ensure solution verifiability by requiring planners to output decomposition traces, avoiding NP-complete verification.

Proposed method

• Introduce a new IPC track for HTN planning with a standardized input language based on a consensus among the community and existing HTN formalisms.
• Adopt a timeline for the track, including domain submission deadlines, preliminary planner submissions, and final competition runs, aligned with the IPC schedule.
• Require all planners to submit both the sequence of primitive actions and the decomposition history to enable efficient and correct solution verification.
• Implement a 'Bring Your Own Benchmark' (BYOB) mode, where each planner submits 20 domain instances, solving at most 10, to ensure balanced and fair domain submissions.
• Use the same technical settings as the classical IPC (1 core, 8 GB RAM, 30-minute time limit) to ensure consistent and comparable performance evaluation.
• Validate preliminary planner submissions early to debug input language compliance and output format correctness before the final competition.

Experimental results

Research questions

• RQ1 How can a standardized input language be established to unify HTN planning systems that currently use incompatible formalisms?
• RQ2 What mechanisms can ensure fair and objective comparison of HTN planners across diverse domains and features?
• RQ3 How can benchmark domains be created that are expressive enough to capture problems undecidable in classical planning, such as Post’s Correspondence Problem or Grammar Intersection?
• RQ4 What role can the 'Bring Your Own Benchmark' (BYOB) model play in promoting balanced, medium-difficulty domain submissions that are not biased toward individual planners?
• RQ5 How can solution verification be made efficient and scalable, given that it is NP-complete without decomposition traces?

Key findings

• The proposed HTN IPC track will establish a common input language and core feature set, enabling interoperability and fair comparison among HTN planners.
• The inclusion of a 'Bring Your Own Benchmark' (BYOB) mode ensures domain submissions are balanced and not biased toward the submitting planner’s strengths.
• Early submission of preliminary planner versions will allow for debugging of input language compliance and output format correctness, reducing technical issues during the final competition.
• Requiring decomposition traces as part of the output ensures that solution verification remains tractable, avoiding NP-completeness in verification.
• The track is expected to lead to a core of robust, reusable HTN planning software, similar to Fast Downward in classical planning, accelerating future research and adoption.
• The competition will highlight weaknesses in current HTN planning approaches, providing concrete targets for future algorithmic improvements.