[Paper Review] Analyzing Language Learned by an Active Question Answering Agent

Christian Buck, Jannis Bulian+5 more|arXiv (Cornell University)|Jan 23, 2018

Topic Modeling12 references3 citations

TL;DR

This paper investigates how an active question answering agent trained via reinforcement learning adapts its language to improve answer quality when interfacing with a black-box QA system. The agent learns to generate less fluent, more IR-like reformulations—emphasizing term re-weighting, stemming, and repetition—demonstrating that it effectively rediscoveres classical information retrieval techniques rather than adopting natural language fluency.

ABSTRACT

We analyze the language learned by an agent trained with reinforcement learning as a component of the ActiveQA system [Buck et al., 2017]. In ActiveQA, question answering is framed as a reinforcement learning task in which an agent sits between the user and a black box question-answering system. The agent learns to reformulate the user's questions to elicit the optimal answers. It probes the system with many versions of a question that are generated via a sequence-to-sequence question reformulation model, then aggregates the returned evidence to find the best answer. This process is an instance of \emph{machine-machine} communication. The question reformulation model must adapt its language to increase the quality of the answers returned, matching the language of the question answering system. We find that the agent does not learn transformations that align with semantic intuitions but discovers through learning classical information retrieval techniques such as tf-idf re-weighting and stemming.

Motivation & Objective

• To understand how language evolves in an active question answering agent trained via reinforcement learning.
• To investigate whether the agent learns human-like natural language or alternative linguistic strategies for improved performance.
• To determine the extent to which the agent's reformulations reflect classical information retrieval techniques.
• To assess the role of fluency versus effectiveness in machine-machine communication within QA systems.

Proposed method

• The agent uses a sequence-to-sequence question reformulation model trained with policy gradient reinforcement learning to optimize F1 score on answers returned by a black-box QA system.
• The reformulator generates multiple question variants, which are scored by an answer ranking model to select the best candidate.
• The system is evaluated on the SearchQA dataset, where original Jeopardy! clues are preprocessed into keyword-like queries.
• Language fluency is measured using a pre-trained language model's negative log probability per token.
• Term frequency (TF), document frequency (DF), and morphological variation (stemming) are analyzed across original, base-NMT, and AQA-generated reformulations.
• Statistical analysis (t-tests) confirms significant differences in linguistic features between reformulation types.

Experimental results

Research questions

• RQ1 How does the language used by an RL-trained active QA agent differ from natural language and from initial paraphrasing models?
• RQ2 To what extent do the agent's reformulations reflect classical information retrieval techniques such as tf-idf re-weighting and stemming?
• RQ3 Why does the agent favor less fluent, repetitive, or morphologically simplified forms over fluent, grammatically correct questions?
• RQ4 Does the absence of fluency incentives in the reward function lead the agent to discover IR-style strategies instead of natural language?
• RQ5 How do the surface forms of the agent’s questions relate to the internal mechanisms of modern deep QA models like BiDAF?

Key findings

• The AQA agent's reformulations are significantly less fluent than both original SearchQA questions and base-NMT paraphrases, as shown by lower language model probabilities (higher negative log probabilities).
• Despite reduced fluency, AQA's top hypothesis outperforms the original SearchQA questions by 2% absolute F1 on the test set, indicating improved effectiveness.
• 99.8% of AQA-generated reformulations start with 'What is name', a pattern absent in both original and base-NMT questions, suggesting a learned strategy to target named entities.
• AQA reformulations show a 12.5% rate of morphological variants, with double the number of words sharing the same stem compared to SearchQA, indicating systematic stemming behavior.
• The agent increases term frequency through repetition (1.2 average TF) and favors lower document frequency (DF) terms, aligning with tf-idf re-weighting principles.
• The agent learns to generate semantically nonsensical surface variants (e.g., 'densey' from 'dense'), which may still be processed effectively by character-based encoders in models like BiDAF.