[Paper Review] Adversarial Training for Community Question Answer Selection Based on Multi-scale Matching

Yang Xiao, Madian Khabsa+6 more|arXiv (Cornell University)|Apr 22, 2018

Topic Modeling39 references4 citations

TL;DR

This paper proposes an adversarial training framework with multi-scale matching for community question answering (CQA) selection, addressing label imbalance by using a generative model to sample challenging negative examples. The method improves classification performance through iterative refinement of a discriminator and generator, achieving state-of-the-art results on SemEval 2017 and strong performance on SemEval 2016 using only textual features.

ABSTRACT

Community-based question answering (CQA) websites represent an important source of information. As a result, the problem of matching the most valuable answers to their corresponding questions has become an increasingly popular research topic. We frame this task as a binary (relevant/irrelevant) classification problem, and present an adversarial training framework to alleviate label imbalance issue. We employ a generative model to iteratively sample a subset of challenging negative samples to fool our classification model. Both models are alternatively optimized using REINFORCE algorithm. The proposed method is completely different from previous ones, where negative samples in training set are directly used or uniformly down-sampled. Further, we propose using Multi-scale Matching which explicitly inspects the correlation between words and ngrams of different levels of granularity. We evaluate the proposed method on SemEval 2016 and SemEval 2017 datasets and achieves state-of-the-art or similar performance.

Motivation & Objective

• To address the severe label imbalance in CQA datasets, where relevant answers constitute only 2.8–9.3% of total samples.
• To improve the robustness and accuracy of CQA classifiers by replacing uniform or random negative sampling with adversarially generated hard negatives.
• To enhance matching capability by explicitly modeling correlations at multiple granularities—words, n-grams, and hierarchical representations.
• To achieve state-of-the-art performance without relying on meta-information such as answer position or user activity.

Proposed method

• A generative model $G$ is trained to produce challenging negative question-answer pairs that are semantically close to relevant ones, increasing gradient signal during discriminator training.
• The discriminator $D$ is trained via REINFORCE to classify pairs as relevant or irrelevant, using rewards based on prediction confidence.
• The generator and discriminator are alternately optimized using REINFORCE, enabling the generator to learn to produce increasingly hard negatives.
• Multi-scale matching is implemented using a deep CNN to extract hierarchical representations from questions and answers, enabling word-to-word, word-to-ngram, and ngram-to-ngram comparisons.
• Matching scores across multiple levels of granularity are aggregated via a feed-forward network to produce a final relevance score.
• The framework avoids reliance on RNNs or attention mechanisms, instead using CNN-based hierarchical feature extraction for efficient multi-granularity matching.

Experimental results

Research questions

• RQ1 Can adversarial training with a learned generator improve CQA selection performance by focusing on hard negative examples?
• RQ2 How does multi-scale matching—comparing words and n-grams at different granularities—affect relevance classification in CQA?
• RQ3 Can a purely text-based model outperform methods that use meta-information such as answer position or user activity?
• RQ4 Does adversarial training provide greater benefit on highly imbalanced datasets like SemEval 2017 compared to more balanced ones like SemEval 2016?

Key findings

• The proposed method achieved the highest MAP (53.38) and MRR (60.64) on the SemEval 2017 dataset, ranking first among all submissions.
• On the SemEval 2016 dataset, the method achieved a MAP of 49.25 and MRR of 54.89, ranking second and outperforming methods that used meta-information.
• Adversarial training significantly improved performance when combined with multi-scale matching, increasing MAP by 4.29 points and MRR by 5.39 points on SemEval 2017.
• The generator successfully produced semantically plausible negative examples that were challenging for the discriminator, demonstrating effective hard example mining.
• Even without meta-information, the model achieved state-of-the-art results, proving the effectiveness of text-only modeling with adversarial training.
• The ablation study showed that adversarial training with the generator (G) alone degraded performance (MAP: 36.31), confirming the necessity of joint training with the discriminator (D).