[Paper Review] ISEEQ: Information Seeking Question Generation using Dynamic Meta-Information Retrieval and Knowledge Graphs

Manas Gaur, Kalpa Gunaratna+2 more|arXiv (Cornell University)|Dec 12, 2021

Topic Modeling4 citations

TL;DR

ISEEQ is a novel framework for generating information-seeking questions (ISQs) from short user queries using dynamic knowledge graph integration and reinforcement learning. It enhances query understanding via knowledge enrichment, retrieves contextually relevant passages, and generates semantically coherent, diverse ISQs that outperform baselines across four datasets and show strong cross-domain transferability.

ABSTRACT

Conversational Information Seeking (CIS) is a relatively new research area within conversational AI that attempts to seek information from end-users in order to understand and satisfy users' needs. If realized, such a system has far-reaching benefits in the real world; for example, a CIS system can assist clinicians in pre-screening or triaging patients in healthcare. A key open sub-problem in CIS that remains unaddressed in the literature is generating Information Seeking Questions (ISQs) based on a short initial query from the end-user. To address this open problem, we propose Information SEEking Question generator (ISEEQ), a novel approach for generating ISQs from just a short user query, given a large text corpus relevant to the user query. Firstly, ISEEQ uses a knowledge graph to enrich the user query. Secondly, ISEEQ uses the knowledge-enriched query to retrieve relevant context passages to ask coherent ISQs adhering to a conceptual flow. Thirdly, ISEEQ introduces a new deep generative-adversarial reinforcement learning-based approach for generating ISQs. We show that ISEEQ can generate high-quality ISQs to promote the development of CIS agents. ISEEQ significantly outperforms comparable baselines on five ISQ evaluation metrics across four datasets having user queries from diverse domains. Further, we argue that ISEEQ is transferable across domains for generating ISQs, as it shows the acceptable performance when trained and tested on different pairs of domains. The qualitative human evaluation confirms ISEEQ-generated ISQs are comparable in quality to human-generated questions and outperform the best comparable baseline.

Motivation & Objective

• To address the open problem of automatically generating high-quality, curiosity-driven information-seeking questions (ISQs) from short user queries in Conversational Information Seeking (CIS).
• To improve ISQ generation by integrating expert-curated knowledge graphs to enrich query semantics and enable better context retrieval.
• To ensure generated ISQs maintain conceptual flow—semantic relations and logical coherence—through a reinforcement learning-based generation strategy.
• To enable cross-domain generalization of ISQ generation without requiring new crowdsourced data collection for each domain.
• To establish a scalable pipeline for automatic dataset creation to accelerate CIS system development.

Proposed method

• ISEEQ uses a knowledge graph to enrich the initial short user query with semantic entities and relations, improving query representation for downstream retrieval.
• It performs dynamic, knowledge-aware passage retrieval from a large corpus using the enriched query to identify contextually relevant passages for question generation.
• A generative adversarial reinforcement learning (ERL) framework is employed to train the ISQ generator, with rewards shaped by conceptual flow and logical coherence.
• The reward model is trained to optimize for semantic relevance, diversity, and logical progression between ISQs, using entailment constraints and fluency metrics.
• The system employs a two-stage training process: first, a T5-based sequence-to-sequence model is fine-tuned on supervised ISQ data; second, it is further refined via reinforcement learning with human-in-the-loop feedback signals.
• Hyperparameters are tuned using Ray, with specific values for α (0.1971), γ (0.12), and a learning rate of 1.17e-5, trained on four V100 GPUs for 100–120 epochs.

Experimental results

Research questions

• RQ1 Can expert-curated knowledge sources like knowledge graphs improve context retrieval and question generation for ISQs?
• RQ2 Can ISEEQ generate ISQs that maintain semantic relations and logical coherence (conceptual flow)?
• RQ3 Can ISEEQ generalize across domains and generate high-quality ISQs for new domains without requiring new crowdsourced data?
• RQ4 How does ISEEQ perform in cross-domain settings where training and testing occur on different domains?
• RQ5 Can the reinforcement learning component effectively guide the generation of diverse, fluent, and contextually coherent ISQs?

Key findings

• ISEEQ significantly outperforms competitive T5-based and fine-tuned baselines across five ISQ evaluation metrics on four diverse datasets, including QAD, QAMR, FBC, and CAsT-19.
• On the QAMR dataset, ISEEQ achieved a ROUGE-L score of 0.74, a F1 score of 0.89, and a conceptual flow score of 77.8%, outperforming the best baseline.
• In cross-domain transfer experiments, ISEEQ-ERL achieved acceptable performance on {QAD-QAMR} and {QAMR-FBC} pairs, even when training data was smaller than test data.
• Human evaluation showed that ISEEQ-generated ISQs were statistically comparable to human-generated questions (S1) and significantly better than the T5-FT CANARD baseline (S3), with no significant difference between S1 and S2.
• The system demonstrated strong transferability, with ISEEQ trained on a narrow domain (FBC) producing high-quality ISQs for a generic domain, indicating potential for zero-shot or few-shot adaptation.
• ANOVA and post-hoc analysis confirmed that both ISEEQ and human-generated ISQs were significantly more diverse and coherent than the baseline, with S1 and S2 showing no statistical difference in mean scores.