[Paper Review] PathVQA: 30000+ Questions for Medical Visual Question Answering

Xuehai He, Yichen Zhang+3 more|arXiv (Cornell University)|Mar 7, 2020

Multimodal Machine Learning Applications33 references85 citations

TL;DR

The paper introduces PathVQA, a pathology VQA dataset with 32,799 QA pairs from 4,998 images, generated semi-automatically from textbooks, and provides baselines with several VQA models.

ABSTRACT

Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a medical VQA dataset is much more challenging. First, due to privacy concerns, pathology images are usually not publicly available. Second, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. To address these challenges, we resort to pathology textbooks and online digital libraries. We develop a semi-automated pipeline to extract pathology images and captions from textbooks and generate question-answer pairs from captions using natural language processing. We collect 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. To our best knowledge, this is the first dataset for pathology VQA. Our dataset will be released publicly to promote research in medical VQA.

Motivation & Objective

• Create a pathology VQA dataset to enable AI to handle ABP-style pathology questions.
• Leverage publicly available pathology textbooks and libraries to bypass privacy and annotation bottlenecks.
• Provide baseline VQA benchmarks with multiple state-of-the-art models for comparison.

Proposed method

• Extract pathology images and captions from online textbooks and PEIR Digital Library.
• Semi-automatically generate question–answer pairs from captions using NLP and a rule-based transducer.
• Manually proofread and post-process generated questions to fix grammar and semantics.

Experimental results

Research questions

• RQ1 Can an AI system answer board-style pathology questions from images and associated captions?
• RQ2 How large and diverse is a pathology VQA dataset that is shareable with the research community?
• RQ3 What baseline performance do current VQA models achieve on open-ended and yes/no pathology questions?

Key findings

• The PathVQA dataset contains 32,799 QA pairs across 4,998 images.
• There are 7 question categories with 49.8% yes/no and 40.9% what-type open-ended questions.
• Baseline results show yes/no accuracy ranging from 57.6% to 68.2% across methods.
• Open-ended QA metrics (BLEU-1/2/3, Exact match, F1) are relatively low (e.g., BLEU-1 up to 32.4, Exact match as low as 2.9%).
• Method 1 with bilinear attention and region proposals achieves the best yes/no performance (68.2%).
• Method 3 with Faster R-CNN further improves over holistic image features (62.0% on yes/no).