[Paper Review] Coarse-to-Fine Q-attention with Learned Path Ranking

Stephen James, Pieter Abbeel|arXiv (Cornell University)|Apr 4, 2022

Robotic Path Planning Algorithms4 citations

TL;DR

This paper introduces Learned Path Ranking (LPR), a method that enhances C2F-ARM by learning to rank diverse paths—generated via planning, Bézier curves, or a behaviorally cloned policy—to enable sample-efficient, vision-based reinforcement learning for complex manipulation tasks requiring precise motions. The key contribution is C2F-ARM+LPR, which achieves 100% success on 16 RLBench tasks and learns real-world tasks like opening a toilet seat or filing cabinet in 10–15 minutes with only 3 demonstrations.

ABSTRACT

We propose Learned Path Ranking (LPR), a method that accepts an end-effector goal pose, and learns to rank a set of goal-reaching paths generated from an array of path generating methods, including: path planning, Bezier curve sampling, and a learned policy. The core idea being that each of the path generation modules will be useful in different tasks, or at different stages in a task. When LPR is added as an extension to C2F-ARM, our new system, C2F-ARM+LPR, retains the sample efficiency of its predecessor, while also being able to accomplish a larger set of tasks; in particular, tasks that require very specific motions (e.g. opening toilet seat) that need to be inferred from both demonstrations and exploration data. In addition to benchmarking our approach across 16 RLBench tasks, we also learn real-world tasks, tabula rasa, in 10-15 minutes, with only 3 demonstrations.

Motivation & Objective

• To address the limitation of C2F-ARM in learning tasks requiring highly specific motions, such as opening a toilet seat, which cannot be solved by generic path planning alone.
• To enable a single agent to learn diverse, task-specific motion patterns without prior kinematic knowledge of objects.
• To improve sample efficiency in real-world training by combining multiple path generation methods with a learned ranking function.
• To allow the system to generalize across tasks with sparse rewards and visual inputs, including those requiring fine-grained motion control.
• To reduce reliance on hand-coded motion primitives by learning path preferences from demonstrations and exploration data.

Proposed method

• LPR learns a Q-function to rank a diverse set of collision-free paths generated from three sources: sampling-based path planning, Bézier curve sampling, and a behaviorally cloned policy.
• The path policy is trained via behavior cloning on paths that successfully complete tasks, enabling it to learn motion preferences without direct reward shaping.
• At inference, the highest-ranked path from the LPR Q-function is selected and executed, allowing the system to dynamically choose between different path generation strategies.
• The method integrates with C2F-ARM by using the next-best pose output as input to the path ranking module, preserving the original system’s sample efficiency.
• The system uses a hybrid path generation pipeline that evolves from sampling-based methods toward policy-based paths as training progresses.
• The path policy is trained without visual input, minimizing computational overhead while still enabling effective path selection.

Experimental results

Research questions

• RQ1 Can a learned path ranking mechanism improve task coverage in vision-based, sparse-reward reinforcement learning without prior object-specific knowledge?
• RQ2 How effective is a hybrid path generation approach combining planning, curve sampling, and a learned policy in solving complex manipulation tasks?
• RQ3 Can LPR enable C2F-ARM to learn real-world tasks requiring precise motions with minimal demonstrations?
• RQ4 Does the integration of a behaviorally cloned path policy improve generalization and success rates on tasks with high motion specificity?
• RQ5 To what extent can the system reduce reliance on sampling-based path planning as training progresses?

Key findings

• C2F-ARM+LPR achieves 100% success rate across 16 RLBench tasks, including those requiring highly specific motions such as opening a toilet seat or drawer.
• The system learns real-world tasks like opening a filing cabinet and a toy dishwasher in approximately 10–15 minutes using only 3 teleoperation demonstrations.
• The path policy is ranked highest in tasks requiring sliding or linear motions (e.g., open_drawer), while curved paths are preferred for revolute joint tasks (e.g., lift_toilet_seat_up).
• The method maintains the sample efficiency of C2F-ARM while expanding its task repertoire, demonstrating no degradation in performance on previously solved tasks.
• The learned path policy becomes increasingly effective as training progresses, reducing dependency on sampling-based path generation.
• Qualitative results show that LPR correctly selects appropriate path types based on task dynamics, with visualizations confirming consistent path selection patterns across episodes.