[Paper Review] Value-of-Information based Arbitration between Model-based and Model-free Control

Krishn Bera, Yash Mandilwar+1 more|arXiv (Cornell University)|Dec 8, 2019

Receptor Mechanisms and Signaling4 citations

TL;DR

This paper proposes a value-of-information (VoI)-based arbitration mechanism that dynamically switches between model-based and model-free reinforcement learning to balance computational efficiency and adaptability. By using uncertainty in Q-value variance as a VoI signal, the agent prioritizes model-based planning when uncertain and reverts to efficient model-free learning as confidence grows, outperforming standard Q-learning and Q-learning with experience replay in skill acquisition tasks.

ABSTRACT

There have been numerous attempts in explaining the general learning behaviours using model-based and model-free methods. While the model-based control is flexible yet computationally expensive in planning, the model-free control is quick but inflexible. The model-based control is therefore immune from reward devaluation and contingency degradation. Multiple arbitration schemes have been suggested to achieve the data efficiency and computational efficiency of model-based and model-free control respectively. In this context, we propose a quantitative 'value of information' based arbitration between both the controllers in order to establish a general computational framework for skill learning. The interacting model-based and model-free reinforcement learning processes are arbitrated using an uncertainty-based value of information. We further show that our algorithm performs better than Q-learning as well as Q-learning with experience replay.

Motivation & Objective

• To address the trade-off between computational cost in model-based planning and sample inefficiency in model-free learning during skill acquisition.
• To develop a biologically plausible, dynamic arbitration mechanism between model-based and model-free controllers using a quantitative cost-benefit analysis.
• To enable the agent to transition from goal-directed, flexible planning to habitual, fast execution as skill proficiency increases.
• To validate the framework against behavioral phenomena in sensorimotor skill learning, particularly Fitts’ three-phase model of learning.

Proposed method

• The agent operates in two stages: 'plan' (model-based) and 'act' (model-free), with arbitration based on a value-of-information (VoI) metric.
• VoI is computed as $ VoI(s,a) = C_{(s,a)} / ( au(s) + ho) $, where $ C_{(s,a)} $ is the variance of Q-values for state-action pair $ (s,a) $, and $ au(s) $ is the variance of Q-values across all actions in state $ s $.
• Model-based planning is triggered only when VoI exceeds a threshold, indicating high uncertainty; otherwise, model-free Q-learning is used.
• Model-based planning uses depth-limited value iteration with forward search, transitioning to off-policy Q-learning when depth is zero.
• The framework learns environment dynamics (transition probabilities) during early training, enabling robustness to reward devaluation and contingency changes.
• The arbitration mechanism evolves over time: model-based control dominates early (200–300 episodes), then gradually yields to model-free control after ~400 episodes.

Experimental results

Research questions

• RQ1 How can a dynamic arbitration mechanism between model-based and model-free reinforcement learning be designed to balance computational cost and learning efficiency?
• RQ2 To what extent does a value-of-information (VoI) metric based on Q-value variance enable optimal switching between planning and reactive behaviors?
• RQ3 Does the proposed framework reproduce the three-phase progression of skill learning (cognitive, associative, motor) observed in behavioral studies?
• RQ4 Can the hybrid model outperform standard Q-learning and Q-learning with experience replay in terms of sample efficiency and convergence speed?

Key findings

• The VoI-based arbitration mechanism successfully transitions from model-based to model-free control over ~400 training episodes, mirroring the shift from goal-directed to habitual behavior in Fitts’ theory.
• Model-based evaluations drop to zero after ~400 episodes, indicating full takeover by the model-free controller, which is computationally efficient and suitable for mature skill execution.
• The agent achieves better performance than standard Q-learning and Q-learning with experience replay, demonstrating improved learning efficiency and convergence.
• The framework learns transition probabilities (world-model) during early training, enabling resilience to reward devaluation and contingency degradation.
• The VoI metric increases with learning progress, reflecting reduced uncertainty and guiding the agent to switch to faster, model-free execution when confidence is high.
• The model-based controller remains dominant in the cognitive phase (first 200–300 episodes), supporting data-efficient, adaptive planning in uncertain environments.