[Paper Review] Dual Supervised Learning

Yingce Xia, Tao Qin+4 more|arXiv (Cornell University)|Jul 3, 2017

Natural Language Processing Techniques72 citations

TL;DR

Dual Supervised Learning (DSL) jointly trains primal and dual tasks by enforcing probabilistic duality, improving performance for translation, image processing, and sentiment analysis.

ABSTRACT

Many supervised learning tasks are emerged in dual forms, e.g., English-to-French translation vs. French-to-English translation, speech recognition vs. text to speech, and image classification vs. image generation. Two dual tasks have intrinsic connections with each other due to the probabilistic correlation between their models. This connection is, however, not effectively utilized today, since people usually train the models of two dual tasks separately and independently. In this work, we propose training the models of two dual tasks simultaneously, and explicitly exploiting the probabilistic correlation between them to regularize the training process. For ease of reference, we call the proposed approach \emph{dual supervised learning}. We demonstrate that dual supervised learning can improve the practical performances of both tasks, for various applications including machine translation, image processing, and sentiment analysis.

Motivation & Objective

• Motivation to leverage intrinsic duality between paired tasks (e.g., A→B vs. B→A) to improve both tasks.
• Formulation of a constrained optimization problem that enforces probabilistic duality between primal and dual models.
• Development of a practical algorithm using a duality-based regularization term solvable via Lagrange multipliers.
• Demonstration of DSL effectiveness across machine translation, image processing, and sentiment analysis.
• Analysis of how DSL acts as data-dependent regularization and its impact on generalization.

Proposed method

• Define primal and dual tasks with conditional distributions P(y|x;θ_xy) and P(x|y;θ_yx).
• Introduce probabilistic duality constraint P(x)P(y|x)=P(y)P(x|y) and relax it with a Lagrangian-based regularization term.
• Minimize a weighted sum of standard losses and the duality regularization term on mini-batches.
• Estimate marginals ŜP(x) and ŜP(y) via language models or class distributions to compute the regularization.
• Train both models jointly using standard optimizers (e.g., SGD, Adam) with lambda hyperparameters controlling the duality regularization.

Experimental results

Research questions

• RQ1 Can exploiting probabilistic duality between paired tasks improve performance for both tasks?
• RQ2 How can the duality between primal and dual models be incorporated into a practical training objective?
• RQ3 What is the impact of the duality regularization on translation quality, image classification/generation, and sentiment analysis?
• RQ4 How do marginal distributions inform and stabilize the duality regularization in DSL?

Key findings

• DSL improves performance on all three application domains: translation (BLEU gains across En↔Fr, En↔De, En↔Zh), image classification (lower error rate), and image generation (lower bits-per-dimension).
• In En↔Fr translation, DSL yields BLEU gains of +2.07 (En→Fr) and +0.86 (Fr→En).
• In En↔De translation, DSL yields BLEU gains of +1.37 (En→De) and +0.12 (De→En).
• In En↔Zh translation, DSL yields BLEU gains of +0.42 (En→Zh MT08), +1.92 (Zh→En MT08), +1.05 (En→Zh MT12), and +1.46 (Zh→En MT12).
• On CIFAR-10, DSL reduced ResNet-110 error from 6.43 to 5.40 and improved PixelCNN++ generation (state-of-the-art bpd 2.93 with ResNet-110).
• In sentiment analysis (IMDB), DSL reduced classification error by 0.90 points and slightly improved perplexity.