[Paper Review] Information Extraction from Scientific Literature for Method Recommendation

Yi Luan|arXiv (Cornell University)|Dec 13, 2018

Topic Modeling4 citations

TL;DR

This paper proposes a semi-supervised framework for scientific method recommendation by extracting scientific terms from unannotated literature and constructing a knowledge graph using co-occurrence, dependency parsing, and Wikipedia-derived relations. It achieves state-of-the-art performance in term extraction and enables link prediction for method-task recommendations with minimal human annotation, validated via MRR and human evaluation.

ABSTRACT

As a research community grows, more and more papers are published each year. As a result there is increasing demand for improved methods for finding relevant papers, automatically understanding the key ideas and recommending potential methods for a target problem. Despite advances in search engines, it is still hard to identify new technologies according to a researcher's need. Due to the large variety of domains and extremely limited annotated resources, there has been relatively little work on leveraging natural language processing in scientific recommendation. In this proposal, we aim at making scientific recommendations by extracting scientific terms from a large collection of scientific papers and organizing the terms into a knowledge graph. In preliminary work, we trained a scientific term extractor using a small amount of annotated data and obtained state-of-the-art performance by leveraging large amount of unannotated papers through applying multiple semi-supervised approaches. We propose to construct a knowledge graph in a way that can make minimal use of hand annotated data, using only the extracted terms, unsupervised relational signals such as co-occurrence, and structural external resources such as Wikipedia. Latent relations between scientific terms can be learned from the graph. Recommendations will be made through graph inference for both observed and unobserved relational pairs.

Motivation & Objective

• To address the challenge of recommending relevant scientific methods to researchers by extracting key terms and relations from vast scientific literature.
• To minimize reliance on expensive human-annotated data by leveraging semi-supervised learning and unsupervised relational signals.
• To construct a scientific knowledge graph using co-occurrence, dependency parsing, and Wikipedia to model latent relations between methods and tasks.
• To enable recommendation of unobserved method-task pairs through graph-based link prediction and multi-path relational modeling.
• To evaluate the system using both automatic metrics (MRR, Hits@k) and human evaluation to ensure practical relevance.

Proposed method

• A semi-supervised neural model is trained on a small amount of annotated data and extended using large-scale unannotated scientific papers to improve scientific term extraction performance.
• Co-occurrence patterns and dependency relations are used as unsupervised relational signals to infer connections between scientific terms.
• External knowledge from Wikipedia is integrated as auxiliary resources to enrich relational signals and improve graph representation.
• A multi-path graph inference model combines evidence from multiple sources (e.g., dependency paths, Wikipedia) using weighted path representations and random walk probabilities.
• The scoring function for link prediction combines path representations and learns shared weights per path length across different resource types.
• Open-ended relations are explored via clustering of latent relational representations, enabling discovery of fine-grained method-task relationships without explicit annotation.

Experimental results

Research questions

• RQ1 Can semi-supervised learning achieve state-of-the-art performance in scientific term extraction with minimal annotated data?
• RQ2 How effectively can co-occurrence and external knowledge sources like Wikipedia be used to construct a scientific knowledge graph with minimal human annotation?
• RQ3 To what extent can graph-based link prediction infer novel method-task relationships not yet published in the literature?
• RQ4 How do multi-path relational signals from diverse sources improve recommendation performance compared to single-source approaches?
• RQ5 Can clustering of latent relational representations uncover meaningful, fine-grained relations between scientific methods and tasks?

Key findings

• The proposed semi-supervised term extraction model achieves state-of-the-art performance on scientific term extraction by leveraging large-scale unannotated papers.
• The knowledge graph construction method effectively integrates co-occurrence, dependency relations, and Wikipedia to model scientific relations with minimal human-annotated data.
• Link prediction using the constructed knowledge graph successfully identifies plausible method-task applications, such as predicting GAN's use in NLP, with a high degree of accuracy.
• The system achieves strong performance on automatic evaluation, with MRR and Hits@k scores indicating effective ranking of relevant method-task pairs.
• Human evaluation confirms the relevance of top-30 recommendations, with high precision and recall, validating the practical utility of the approach.
• Clustering of latent relational representations reveals meaningful, fine-grained relations such as method combinations (e.g., HMM + Viterbi) and task similarities (e.g., POS tagging and NER), suggesting potential for open relation learning.