[Paper Review] Analysis of Points of Interests Recommended for Leisure Walk Descriptions

Payal Bajaj, Daniel Campos+2 more|arXiv (Cornell University)|Oct 10, 2024

Topic Modeling31 references1,286 citations

TL;DR

This paper introduces MS MARCO, a large-scale, real-world machine reading comprehension dataset derived from anonymized Bing search queries, containing 1 million questions, 8.8 million passages, and 182,669 human-generated answers. The dataset enables three benchmark tasks—answerability prediction, answer generation, and passage ranking—demonstrating that models like BiDAF and memory networks achieve strong performance, with human-ensemble ROUGE-L scores reaching 0.737 on the novice task, highlighting the dataset's value for evaluating robust MRC and neural IR systems.

ABSTRACT

Data for Sub-Task 1 of the Advertisement in Retrieval-Augmented Generation task at Touché 2025. The dataset contains segments retrieved from the segmented version of MS MARCO V2.1. The queries used in retrieval are taken from the Webis Generated Native Ads 2024 dataset.

Motivation & Objective

• To address the limitations of existing MRC datasets by creating a large-scale, real-world dataset derived from actual user search queries instead of synthetic or crowd-sourced questions.
• To improve the robustness of MRC models by including noisy, real-world text, conflicting information, and unanswerable questions that reflect real user search behavior.
• To support multi-passage and multi-document reasoning by providing context passages from diverse web documents, enabling models to extract and synthesize information across multiple sources.
• To establish a benchmark for three distinct but related MRC tasks: answerability prediction, answer generation, and passage ranking, with increasing levels of difficulty.
• To provide a realistic testbed for evaluating neural IR and MRC models, including their ability to handle out-of-vocabulary terms, typos, and complex reasoning across multiple documents.

Proposed method

• The dataset is constructed from 1,010,916 anonymized Bing search queries, each paired with 182,669 human-generated answers and 8,841,823 context passages extracted from 3.56 million web documents retrieved by Bing.
• Crowd-sourced editors generate answers based on the retrieved passages, are allowed to mark questions as unanswerable, and are encouraged to produce complete, well-formed sentences.
• The dataset supports three tasks: (i) predict answerability and extract/synthesize answers from context passages; (ii) generate a well-formed answer from context; (iii) rank retrieved passages given a question.
• Baseline models include a DSSM-like passage ranking model, a vanilla sequence-to-sequence model, a memory network with attention, and a BiDAF model fine-tuned on span prediction.
• Evaluation uses ROUGE-L and BLEU metrics, with human-ensemble baselines established using top-performing editors on a subset of 1,427 questions.
• The v2.1 dataset includes a revised evaluation protocol with separate novice and intermediate tasks, and models are evaluated on both original and updated data versions.

Experimental results

Research questions

• RQ1 Can MRC models effectively identify whether a question is answerable given a set of context passages, especially when information is fragmented or conflicting?
• RQ2 How well can neural models generate fluent, well-formed answers from retrieved passages when the answer is not a simple span but requires synthesis?
• RQ3 To what extent do passage ranking models trained on real user queries outperform baselines in retrieving relevant documents for complex questions?
• RQ4 How does the inclusion of unanswerable questions and noisy, real-world text affect model generalization and robustness compared to synthetic or clean datasets?
• RQ5 How do model performance and evaluation metrics (e.g., ROUGE-L, BLEU) change when moving from synthetic datasets like SQuAD to real-world datasets like MS MARCO?

Key findings

• The human-ensemble baseline achieved a ROUGE-L score of 0.737 on the novice task and 0.630 on the intermediate task, significantly outperforming machine models, indicating the difficulty of the benchmark.
• The BiDAF model fine-tuned on MS MARCO achieved a ROUGE-L score of 0.268 on the original dataset, but performance dropped to 0.150 on the v2.1 novice task and 0.170 on the intermediate task, indicating increased difficulty in the updated version.
• The best passage baseline achieved a ROUGE-L score of 0.351, showing that even selecting the most relevant passage does not yield a strong answer, underscoring the need for synthesis.
• The passage ranking model achieved a ROUGE-L score of 0.177, significantly lower than the human baseline, indicating room for improvement in neural IR for MRC.
• On the cloze-style numeric subset, ReasoNet achieved 74.7% accuracy on the CNN/Daily Mail test set and 58.9% on the MS MARCO subset, outperforming AS Reader, demonstrating the value of dynamic reasoning in attention mechanisms.
• The memory network model achieved a BLEU score of 0.340 and pa-BLEU of 0.341 on a multi-answer subset, showing that memory-augmented networks can handle complex answer generation, though still below human performance.