[Paper Review] Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach

Jingjing Xu, Xu Sun+5 more|arXiv (Cornell University)|May 14, 2018

Topic ModelingComputer Science12 references58 citations

TL;DR

The paper introduces a cycled reinforcement learning framework with a neutralization module and an emotionalization module to perform sentiment-to-sentiment translation on unpaired data, achieving state-of-the-art content preservation and competitive sentiment transfer.

ABSTRACT

The goal of sentiment-to-sentiment "translation" is to change the underlying sentiment of a sentence while keeping its content. The main challenge is the lack of parallel data. To solve this problem, we propose a cycled reinforcement learning method that enables training on unpaired data by collaboration between a neutralization module and an emotionalization module. We evaluate our approach on two review datasets, Yelp and Amazon. Experimental results show that our approach significantly outperforms the state-of-the-art systems. Especially, the proposed method substantially improves the content preservation performance. The BLEU score is improved from 1.64 to 22.46 and from 0.56 to 14.06 on the two datasets, respectively.

Motivation & Objective

• Motivate sentiment-to-sentiment translation as a form of style transfer with content preservation.
• Overcome lack of parallel data by proposing a cycled reinforcement learning setup.
• Explicitly separate sentiment from content via a neutralization module to enhance translation quality.
• Pre-train modules with a self-attention based sentiment classifier to bootstrap learning.
• Demonstrate superior content preservation on Yelp and Amazon review datasets.

Proposed method

• Two-module architecture: a neutralization module that extracts non-emotional content by filtering emotional words, and an emotionalization module that injects target sentiment into the neutral content.
• Pre-train both modules using supervised objectives; use a self-attention based sentiment classifier to guide neutralization.
• Emotionalization uses a bi-decoder seq2seq framework to generate text conditioned on target sentiment.
• Cycled reinforcement learning: train neutralization with policy gradients using rewards from sentiment accuracy and BLEU-based content preservation, guided by the emotionalization output.
• Rewards combine sentiment confidence and BLEU via a harmonic mean to guide learning (R = (1+β^2) * BLEU * Confid / (β^2 * BLEU + Confid)).
• Utilize unpaired data (reviews with sentiment labels) to train without parallel sentence pairs.

Experimental results

Research questions

• RQ1 Can unpaired sentiment data be used to perform sentiment-to-sentiment translation while preserving content?
• RQ2 Does explicitly separating sentiment from content via neutralization improve content preservation and overall translation quality?
• RQ3 How effective is cycled reinforcement learning in jointly training the neutralization and emotionalization modules?
• RQ4 What is the impact of pre-training with a self-attention sentiment classifier on the final performance?
• RQ5 How does the proposed method compare to state-of-the-art baselines on Yelp and Amazon datasets?

Key findings

• Significant improvement in content preservation compared to baselines (BLEU scores substantially higher).
• Automatic evaluation shows competitive sentiment transfer accuracy with strong content preservation (G-score superior on Yelp and Amazon).
• Human evaluation confirms better semantic preservation with the proposed method over baselines.
• BLEU scores improved from 1.64 to 22.46 (Yelp) and 0.56 to 14.06 (Amazon) for the proposed method.
• The proposed method achieves best overall performance (G-score) among evaluated systems on both datasets.
• The neutralization module effectively filters emotional words, enabling the emotionalization module to better add the target sentiment.