[Paper Review] Generalization Tower Network: A Novel Deep Neural Network Architecture for Multi-Task Learning

Yuhang Song, Main Xu+2 more|arXiv (Cornell University)|Oct 27, 2017

Reinforcement Learning in Robotics21 references3 citations

TL;DR

This paper proposes the Generalization Tower Network (GTN), a novel deep neural network architecture for multi-task reinforcement learning (MT-RL) that integrates horizontal streams for shared feature extraction and vertical streams to model hierarchical shared knowledge across tasks. GTN achieves state-of-the-art performance on 51 Atari games, improving relative final scores by up to 23% over MT-A3C, especially in high-complexity task categories like shooting and adventure games.

ABSTRACT

Deep learning (DL) advances state-of-the-art reinforcement learning (RL), by incorporating deep neural networks in learning representations from the input to RL. However, the conventional deep neural network architecture is limited in learning representations for multi-task RL (MT-RL), as multiple tasks can refer to different kinds of representations. In this paper, we thus propose a novel deep neural network architecture, namely generalization tower network (GTN), which can achieve MT-RL within a single learned model. Specifically, the architecture of GTN is composed of both horizontal and vertical streams. In our GTN architecture, horizontal streams are used to learn representation shared in similar tasks. In contrast, the vertical streams are introduced to be more suitable for handling diverse tasks, which encodes hierarchical shared knowledge of these tasks. The effectiveness of the introduced vertical stream is validated by experimental results. Experimental results further verify that our GTN architecture is able to advance the state-of-the-art MT-RL, via being tested on 51 Atari games.

Motivation & Objective

• Address the limitation of conventional deep neural networks in learning representations for multi-task reinforcement learning (MT-RL), where tasks may require different kinds of shared representations.
• Overcome the need to store and manually select from multiple pre-trained models in existing MT-RL approaches, such as progressive networks.
• Enable a single unified model to master multiple tasks by encoding hierarchical shared knowledge across tasks.
• Improve multi-task generalization performance in MT-RL by introducing a vertical stream architecture that captures task-specific and shared knowledge at different levels of abstraction.

Proposed method

• Design a dual-stream architecture: horizontal streams process high-dimensional inputs using convolutional and LSTM layers to extract multi-level representations.
• Introduce vertical streams that share layers hierarchically across horizontal streams, enabling the model to learn hierarchical shared knowledge across tasks.
• Structure vertical streams such that lower levels encode common knowledge (e.g., 'shoot continuously' in shooting games), while deeper levels encode more specific knowledge (e.g., 'aim at valid targets and shoot').
• Train the GTN model end-to-end using an asynchronous advantage actor-critic (A3C) framework, with shared weights across vertical stream layers to enforce knowledge transfer.
• Normalize step rewards per game during training to ensure fair comparison across tasks with varying score scales.
• Evaluate performance using Relative Final Score (RFS), defined as the ratio of multi-task agent score to single-task agent score, to measure generalization capability across tasks.

Experimental results

Research questions

• RQ1 Can a single deep neural network architecture effectively generalize across multiple diverse reinforcement learning tasks by modeling hierarchical shared knowledge?
• RQ2 How does the inclusion of vertical streams for hierarchical knowledge encoding improve multi-task generalization compared to standard horizontal-only architectures?
• RQ3 To what extent does the GTN architecture outperform existing MT-RL baselines like MT-A3C in terms of performance and scalability across a large set of tasks?
• RQ4 Does the GTN architecture show particular advantages in scenarios with a high number of similar tasks, such as in shooting or adventure games on the Atari platform?

Key findings

• The GTN architecture achieves a 65% average Relative Final Score (RFS) on all shooting games when using 4 vertical stream levels (M=4) and 4 horizontal layers (N=4), significantly outperforming baseline methods.
• The vertical stream contributes substantially to multi-task generalization, improving RFS from 52% (M=1) to 65% (M=4) at N=4, while horizontal layers have minimal impact on multi-task performance.
• GTN improves performance over MT-A3C by up to 23% in the shooting games category, with consistent gains across all 6 Atari game categories.
• The horizontal stream is most effective for single-task learning, achieving 104% RFS at N=4 and M=2, indicating its strength in feature extraction per task.
• The performance gap between GTN and MT-A3C is most pronounced in high-complexity, high-task-count categories like shooting and adventure games, demonstrating GTN’s scalability and generalization strength.
• The model generalizes well across diverse tasks without requiring task labels or manual model selection, enabling end-to-end multi-task learning in a single unified architecture.