[Paper Review] Hyper-Decision Transformer for Efficient Online Policy Adaptation

Mengdi Xu, Yuchen Lu+4 more|arXiv (Cornell University)|Apr 17, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Hyper-Decision Transformer (HDT), a parameter-efficient framework that enables fast online adaptation of pre-trained Decision Transformers to novel tasks using only a few demonstrations. By using a hyper-network to conditionally initialize adapter modules within the transformer, HDT fine-tunes just 0.5% of parameters and achieves faster convergence and superior success rates than full fine-tuning or prompting baselines, even without expert actions.

ABSTRACT

Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin.

Motivation & Objective

• To address the challenge of efficiently adapting pre-trained Decision Transformers to unseen tasks with minimal data and computation.
• To improve generalization in both meta-imitation learning (with expert actions) and meta-Learning-from-Observation (without expert actions).
• To achieve strong performance with minimal parameter updates, maintaining efficiency while outperforming full fine-tuning and prompting methods.
• To explore the scalability and robustness of the method under varying model sizes and training task diversity.

Proposed method

• HDT integrates a pre-trained Decision Transformer with a lightweight adapter module inserted into each transformer block, which is fine-tuned during adaptation.
• A hyper-network generates the initial parameters of the adapter module based on a few demonstrations, encoding task-specific information.
• The adapter uses a bottleneck structure to minimize parameter overhead, with only 0.5% of the base model's parameters updated during adaptation.
• In meta-IL, the adapter is fine-tuned in a supervised manner using expert demonstrations; in meta-LfO, it is updated via online reinforcement learning using only states.
• The hyper-network is trained on diverse tasks to generalize to unseen tasks, enabling effective initialization without requiring expert actions.
• The framework supports both few-shot imitation learning and zero-shot transfer to unseen environments with strong data efficiency.

Figure 1: Efficient online policy adaptation of pre-trained transformer models with few-shot demonstrations. To facilitate data efficiency, we introduce a demonstration-conditioned adaptation module that helps leverage prior knowledge in the demonstration and guide exploration. When adapting to nove

Experimental results

Research questions

• RQ1 Can a hyper-network effectively initialize adapter parameters to enable fast and efficient policy adaptation in unseen tasks?
• RQ2 How does HDT's performance compare to full fine-tuning and prompting-based methods in terms of data and parameter efficiency?
• RQ3 Can HDT generalize to novel tasks when expert actions are unavailable, i.e., in the meta-LfO setting?
• RQ4 How does model size and training task diversity affect the generalization and adaptation performance of HDT?
• RQ5 Does the hyper-network successfully extract and encode meaningful task-specific information into the adapter parameters?

Key findings

• With only one expert demonstration and fine-tuning just 0.5% of the model parameters, HDT adapts faster to unseen tasks than full fine-tuning of the Decision Transformer.
• In the meta-LfO setting without expert actions, HDT outperforms state-of-the-art baselines in success rates by a large margin, demonstrating strong generalization.
• The hyper-network successfully encodes task-specific information, as evidenced by distinguishable adapter parameter distributions across different tasks in visualization.
• Reducing the adapter’s bottleneck size or the base DT’s model size leads to slower convergence and performance degradation, especially in the meta-LfO setting.
• HDT shows improved success rates and faster convergence with increased training task diversity and larger model sizes, confirming scalability.
• HDT significantly outperforms prompt-tuning and IA3-based fine-tuning baselines in both meta-IL and meta-LfO, particularly in success rate and data efficiency.

Figure 2: Model architecture of Hyper-Decision Transformer (HDT). Similar to DT, HDT takes recent contexts as input and outputs fine-grind actions. To encode task-specific information, HDT injects adapter layers into each decoder block. The adapter layer’s parameters come from a stand-alone hyper-ne