[Paper Review] Learning to Search Better Than Your Teacher

Kai-Wei Chang, Akshay Krishnamurthy+3 more|arXiv (Cornell University)|Feb 8, 2015

Advanced Bandit Algorithms ResearchDecision Sciences19 references91 citations

TL;DR

This paper introduces LOLS (Locally Optimal Learning to Search), a novel learning-to-search algorithm that guarantees low regret not only relative to a potentially suboptimal reference policy but also relative to one-step deviations from the learned policy. Unlike prior methods that only match the reference policy’s performance, LOLS ensures local optimality, enabling significant improvements over poor baselines in structured prediction and structured contextual bandit settings.

ABSTRACT

Methods for learning to search for structured prediction typically imitate a reference policy, with existing theoretical guarantees demonstrating low regret compared to that reference. This is unsatisfactory in many applications where the reference policy is suboptimal and the goal of learning is to improve upon it. Can learning to search work even when the reference is poor? We provide a new learning to search algorithm, LOLS, which does well relative to the reference policy, but additionally guarantees low regret compared to deviations from the learned policy: a local-optimality guarantee. Consequently, LOLS can improve upon the reference policy, unlike previous algorithms. This enables us to develop structured contextual bandits, a partial information structured prediction setting with many potential applications.

Motivation & Objective

• Address the limitation of existing learning-to-search methods that only guarantee performance relative to a reference policy, which may be suboptimal.
• Develop a learning-to-search algorithm that ensures local optimality even when the reference policy is poor.
• Enable effective learning in structured contextual bandit settings where full supervision is unavailable and the goal is to improve upon an existing (possibly suboptimal) rule-based system.
• Provide theoretical guarantees that balance regret relative to the reference policy and regret relative to one-step policy deviations.
• Demonstrate the superiority of LOLS over prior algorithms in settings where local hill-climbing in policy space is effective but the reference policy is far from optimal.

Proposed method

• Propose LOLS, an online learning-to-search algorithm that maintains a convex combination of regret-to-reference and regret-to-own-one-step-deviations.
• Use a general algorithmic schematic that generalizes prior methods like Searn, DAgger, and AggreVaTe, enabling compatibility with existing search-based structured prediction frameworks.
• Incorporate cost-sensitive classification in mini-batches to update the policy, ensuring convergence to locally optimal policies.
• Introduce a regret decomposition that separates performance relative to the reference policy and relative to local policy improvements.
• Apply a combinatorial analysis of the hypercube graph to bound the number of policy updates required to reach local optimality, leveraging the snake-in-the-box problem.
• Extend LOLS to structured contextual bandits by modifying the learning rule to handle partial feedback and maintaining the same regret guarantees.

Experimental results

Research questions

• RQ1 Can learning-to-search methods improve beyond a suboptimal reference policy, rather than merely matching it?
• RQ2 What theoretical guarantees can be provided when the reference policy is not optimal, especially in terms of local optimality?
• RQ3 How does the performance of LOLS compare to prior learning-to-search algorithms when the reference policy is poor but local improvements are possible?
• RQ4 Can LOLS be effectively adapted to the structured contextual bandit setting, where feedback is partial and the goal is to improve upon an existing system?
• RQ5 What is the worst-case number of policy updates required for LOLS to reach a locally optimal policy, and how does this scale with the search space?

Key findings

• LOLS achieves a regret bound that combines regret relative to the reference policy and regret relative to one-step policy deviations, ensuring local optimality even when the reference is poor.
• Theoretical analysis shows that LOLS outperforms prior algorithms when the reference policy is suboptimal but local hill-climbing is effective.
• In the structured contextual bandit setting, LOLS provides a natural extension with corresponding regret guarantees, enabling learning from partial feedback.
• Empirical results confirm that LOLS significantly outperforms the reference policy on real-world datasets, demonstrating practical superiority.
• The worst-case number of policy updates required for LOLS to reach local optimality is bounded by the length of the longest path in a hypercube (snake-in-the-box), which is Θ(2^T), showing scalability limitations in high-dimensional policy spaces.
• The algorithm is forced to follow one-step deviations due to cost function structure, even when processing mini-batches, ensuring convergence to local optima without escaping the path.