[Paper Review] Natural Language Understanding with the Quora Question Pairs Dataset

Lakshay Sharma, Laura Graesser+2 more|arXiv (Cornell University)|Jul 1, 2019

Topic ModelingComputer Science20 references57 citations

TL;DR

The paper studies natural language understanding through duplicate question detection on the Quora dataset, finding that a simple Continuous Bag of Words model outperforms more complex recurrent/attention models, with noted labeling subjectivity.

ABSTRACT

This paper explores the task Natural Language Understanding (NLU) by looking at duplicate question detection in the Quora dataset. We conducted extensive exploration of the dataset and used various machine learning models, including linear and tree-based models. Our final finding was that a simple Continuous Bag of Words neural network model had the best performance, outdoing more complicated recurrent and attention based models. We also conducted error analysis and found some subjectivity in the labeling of the dataset.

Motivation & Objective

• Investigate Natural Language Understanding via duplicate question detection on the Quora dataset.
• Assess a range of machine learning models from linear to tree-based and neural architectures.
• Identify which modeling approach yields best performance on this NLU task.
• Perform error analysis to understand labeling subjectivity and dataset limitations.

Proposed method

• Experiment with linear, tree-based, and neural network models on the Quora duplicate question task.
• Apply a Continuous Bag of Words (CBOW) neural network as a baseline and compare against recurrent and attention-based models.
• Conduct empirical evaluations to assess model performance on the dataset.
• Perform error analysis to examine labeling subjectivity in the Quora dataset.

Experimental results

Research questions

• RQ1 Which machine learning model family (linear, tree-based, neural) provides the best performance for Quora duplicate question detection?
• RQ2 Does a simple CBOW model outperform more complex recurrent/attention-based models on this task?
• RQ3 What labeling issues or subjectivity affect the Quora dataset and model evaluation?

Key findings

• A simple Continuous Bag of Words neural network achieved the best performance among the explored models.
• More complex recurrent and attention-based models did not outperform CBOW on this task.