[Paper Review] Learning to Switch Between Machines and Humans.

Vahid Balazadeh Meresht, Abir De+2 more|arXiv (Cornell University)|Feb 11, 2020

Reinforcement Learning in Robotics49 references8 citations

TL;DR

This paper proposes a reinforcement learning algorithm that learns to switch control between human and machine agents in semi-autonomous systems using a two-layer Markov decision process. It achieves sublinear regret by leveraging upper confidence bounds on policies and environment transitions, outperforming generic algorithms in obstacle avoidance simulations.

ABSTRACT

Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions. In this work, our goal is to develop algorithms that, by learning to switch control between machine and human agents, allow existing reinforcement learning agents to operate under different automation levels. To this end, we first formally define the problem of learning to switch control among agents in a team via a 2-layer Markov decision process. Then, we develop an online learning algorithm that uses upper confidence bounds on the agents' policies and the environment's transition probabilities to find a sequence of switching policies. We prove that the total regret of our algorithm with respect to the optimal switching policy is sublinear in the number of learning steps. Moreover, we also show that our algorithm can be used to find multiple sequences of switching policies across several independent teams of agents operating in similar environments, where it greatly benefits from maintaining shared confidence bounds for the environments' transition probabilities. Simulation experiments in obstacle avoidance in a semi-autonomous driving scenario illustrate our theoretical findings and demonstrate that, by exploiting the specific structure of the problem, our proposed algorithm is superior to problem-agnostic algorithms.

Motivation & Objective

• Address the gap in reinforcement learning where agents are assumed fully autonomous, despite real-world systems requiring human-machine collaboration.
• Formally define a two-layer Markov decision process for learning optimal switching policies between human and machine agents.
• Develop an online learning algorithm that adapts switching policies using confidence bounds on policies and environment dynamics.
• Demonstrate the algorithm’s scalability across multiple independent agent teams by sharing confidence bounds on environment transitions.
• Validate the approach in a semi-autonomous driving scenario with obstacle avoidance, showing superiority over problem-agnostic methods.

Proposed method

• Model the switching control problem as a two-layer MDP, where the top layer selects switching policies and the bottom layer governs agent actions.
• Use upper confidence bounds (UCB) on both agents' policies and the environment’s transition probabilities to balance exploration and exploitation.
• Maintain shared confidence bounds across multiple independent teams to improve sample efficiency and reduce computational overhead.
• Design an online learning algorithm that dynamically updates switching policies based on observed outcomes and uncertainty estimates.
• Prove theoretical sublinear regret with respect to the optimal switching policy, ensuring long-term performance convergence.
• Integrate domain-specific structural assumptions in the obstacle avoidance task to enhance learning efficiency.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent learn to switch control between human and machine agents to improve performance in semi-autonomous systems?
• RQ2 How can confidence bounds on policies and environment transitions be used to minimize regret in dynamic switching scenarios?
• RQ3 What is the impact of sharing confidence bounds across multiple independent agent teams on learning efficiency and performance?
• RQ4 Does the proposed algorithm outperform generic reinforcement learning methods in structured, real-world tasks like obstacle avoidance?
• RQ5 To what extent does exploiting problem structure improve learning speed and accuracy compared to general-purpose algorithms?

Key findings

• The proposed algorithm achieves sublinear regret with respect to the optimal switching policy, indicating long-term convergence to near-optimal performance.
• The algorithm significantly outperforms problem-agnostic reinforcement learning baselines in obstacle avoidance simulations within a semi-autonomous driving environment.
• Sharing confidence bounds across multiple independent teams improves learning efficiency and reduces overall regret.
• The use of structured confidence bounds on policies and environment transitions enables faster convergence and better decision-making in complex tasks.
• Simulation results confirm that leveraging domain-specific problem structure enhances algorithmic performance beyond generic RL approaches.