[Paper Review] Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning

Ziyu Yao, Xiujun Li+3 more|arXiv (Cornell University)|Aug 21, 2018

Software Engineering Research41 references3 citations

TL;DR

This paper proposes a hierarchical reinforcement learning (HRL) agent for interactive semantic parsing of If-Then recipes, where the agent dynamically selects clarification questions and subtask order to improve parsing accuracy with minimal user interaction. The HRL framework outperforms non-interactive and rule-based baselines in both simulation and human evaluation, achieving higher accuracy while reducing the number of required questions.

ABSTRACT

Given a text description, most existing semantic parsers synthesize a program in one shot. However, it is quite challenging to produce a correct program solely based on the description, which in reality is often ambiguous or incomplete. In this paper, we investigate interactive semantic parsing, where the agent can ask the user clarification questions to resolve ambiguities via a multi-turn dialogue, on an important type of programs called "If-Then recipes." We develop a hierarchical reinforcement learning (HRL) based agent that significantly improves the parsing performance with minimal questions to the user. Results under both simulation and human evaluation show that our agent substantially outperforms non-interactive semantic parsers and rule-based agents.

Motivation & Objective

• To address the challenge of parsing ambiguous or incomplete natural language descriptions for If-Then recipes, which often lead to incorrect program synthesis in one-shot parsing.
• To reduce user interaction cost by minimizing the number of clarification questions while maximizing parsing accuracy through intelligent question selection.
• To develop a reinforcement learning framework that learns when and what to ask during interactive parsing, using only final correctness feedback as supervision.
• To decompose the complex parsing task into four subtasks—trigger channel, trigger function, action channel, and action function—using hierarchical policy learning.
• To enable the agent to dynamically optimize the order of subtask execution based on context and user responses, improving overall efficiency and accuracy.

Proposed method

• Formulate interactive semantic parsing as a hierarchical Markov Decision Process (MDP) with four predefined subtasks: predicting trigger and action components.
• Implement a two-level policy: a high-level policy that selects the optimal subtask order, and low-level policies that handle each subtask by deciding whether to ask a question or predict a component.
• Train the policies using sparse reward signals—only whether each predicted component is correct—without requiring explicit supervision on question timing or user response quality.
• Use options in the HRL framework to enable temporal abstraction, reducing state-action space complexity and improving learning efficiency.
• Design a user simulator to generate synthetic user responses during training, enabling end-to-end training without real human interaction.
• Optimize for both high parsing accuracy and low question count by balancing exploration and exploitation in the reward function.

Experimental results

Research questions

• RQ1 Can a hierarchical reinforcement learning agent effectively resolve ambiguity in If-Then recipe descriptions through interactive clarification questions?
• RQ2 How does the agent’s ability to dynamically choose subtask order affect parsing accuracy and interaction efficiency?
• RQ3 Can an HRL-based agent outperform non-interactive and rule-based agents in both simulated and real human evaluation settings?
• RQ4 To what extent can the agent minimize the number of questions while maintaining high accuracy, given only correctness feedback as supervision?
• RQ5 How generalizable is the HRL framework to other semantic parsing tasks with naturally decomposable subtasks?

Key findings

• The HRL-based agent significantly outperforms non-interactive semantic parsers in both simulation and human evaluation, achieving higher parsing accuracy on ambiguous descriptions.
• The HRL agent asks fewer questions than the HRL-fixedOrder baseline by intelligently coordinating subtask order, reducing interaction cost without sacrificing accuracy.
• In human evaluation, the HRL agent successfully parses ambiguous descriptions like 'record to evernote' into the correct If-Then recipe by asking contextually relevant questions.
• The agent learns to ask only necessary questions through reinforcement learning, using only correctness feedback as supervision, without requiring labeled question timing data.
• The HRL framework generalizes well to other semantic parsing tasks, such as knowledge-based question answering, due to its modular, subtask-decomposed structure.
• The agent’s performance is robust even when user responses are in natural language, demonstrating effective handling of real-world user input variability.