[Paper Review] Information Aggregation via Dynamic Routing for Sequence Encoding

Jingjing Gong, Xipeng Qiu+2 more|arXiv (Cornell University)|Jun 5, 2018

Text and Document Classification Technologies44 citations

TL;DR

The paper introduces a dynamic routing-based aggregation (DR-AGG) mechanism to compress variable-length sequence encodings into fixed-size vectors, improving text classification performance over pooling and self-attention across five datasets. It presents standard and reversed routing policies and demonstrates superior accuracy on document- and sentence-level tasks.

ABSTRACT

While much progress has been made in how to encode a text sequence into a sequence of vectors, less attention has been paid to how to aggregate these preceding vectors (outputs of RNN/CNN) into fixed-size encoding vector. Usually, a simple max or average pooling is used, which is a bottom-up and passive way of aggregation and lack of guidance by task information. In this paper, we propose an aggregation mechanism to obtain a fixed-size encoding with a dynamic routing policy. The dynamic routing policy is dynamically deciding that what and how much information need be transferred from each word to the final encoding of the text sequence. Following the work of Capsule Network, we design two dynamic routing policies to aggregate the outputs of RNN/CNN encoding layer into a final encoding vector. Compared to the other aggregation methods, dynamic routing can refine the messages according to the state of final encoding vector. Experimental results on five text classification tasks show that our method outperforms other aggregating models by a significant margin. Related source code is released on our github page.

Motivation & Objective

• Motivate and improve how to aggregate word-level encodings into a fixed-size text representation.
• Propose a dynamic routing aggregation mechanism inspired by Capsule Networks.
• Compare dynamic routing with pooling and self-attention across standard and reversed policies.
• Demonstrate scalability to long texts via hierarchical routing and validate on multiple datasets.

Proposed method

• Encode words with a BiLSTM to obtain phrase-level representations.
• Introduce dynamic routing (DR-AGG) to transfer information from input capsules to output capsules.
• Compute messages m_{i→j} via c_{ij}f(h_i, θ_j) and update output capsules v_j with a squash nonlinearity.
• Iteratively refine routing coefficients c_{ij} using agreement between v_j and f(h_i, θ_j).
• Concatenate output capsules to form the fixed-size encoding e for prediction.
• Explore a reversed DR-AGG where output capsules guide the reception of information from input capsules.
• Apply hierarchical dynamic routing to handle long documents by routing at word and sentence levels.

Experimental results

Research questions

• RQ1 How can we effectively aggregate variable-length word encodings into a fixed-size sequence representation?
• RQ2 Does a dynamic routing policy improve information transfer compared to pooling or self-attention in text classification?
• RQ3 What is the effect of standard versus reversed dynamic routing on performance?
• RQ4 Can hierarchical DR-AGG efficiently scale to long documents without sacrificing accuracy?

Key findings

• DR-AGG outperforms max pooling, average pooling, and self-attention across five datasets.
• On document-level datasets Yelp-2013, Yelp-2014, and IMDB, DR-AGG improves over the best previous models by 2.5%, 3.0%, and 1.6%, respectively.
• On SST-1 and SST-2, DR-AGG also achieves superior results compared to the baselines.
• Standard DR-AGG generally outperforms Reverse DR-AGG, indicating effective information routing from inputs to outputs.
• Routing iterations around 3 yield peak performance across various capsule configurations.
• Visualization shows different capsules specialize on different sentence aspects, reducing information redundancy.