[Paper Review] A Scientific Information Extraction Dataset for Nature Inspired Engineering

Ruben Kruiper, Julian F. V. Vincent+3 more|arXiv (Cornell University)|May 15, 2020

Topic Modeling52 references4 citations

TL;DR

This paper introduces FOBIE, a dataset of 1,500 manually-annotated scientific biology sentences focused on extracting domain-independent relations—particularly trade-offs—between key biological concepts. The dataset enables training and evaluation of relation extraction models, with a state-of-the-art neural model (SciIE) achieving strong performance, especially on trade-off relations, despite challenges from syntactic variation and long sentences.

ABSTRACT

Nature has inspired various ground-breaking technological developments in applications ranging from robotics to aerospace engineering and the manufacturing of medical devices. However, accessing the information captured in scientific biology texts is a time-consuming and hard task that requires domain-specific knowledge. Improving access for outsiders can help interdisciplinary research like Nature Inspired Engineering. This paper describes a dataset of 1,500 manually-annotated sentences that express domain-independent relations between central concepts in a scientific biology text, such as trade-offs and correlations. The arguments of these relations can be Multi Word Expressions and have been annotated with modifying phrases to form non-projective graphs. The dataset allows for training and evaluating Relation Extraction algorithms that aim for coarse-grained typing of scientific biological documents, enabling a high-level filter for engineers.

Motivation & Objective

• To address the challenge of accessing relevant biological information for engineers in nature-inspired engineering, where domain-specific terminology and conceptual differences hinder discovery.
• To develop a dataset that supports the extraction of high-level, domain-independent relations—especially trade-offs—from scientific biology texts to enable cross-domain literature discovery.
• To provide a resource for training and evaluating relation extraction models that can identify complex, non-projective relations involving multi-word expressions and modifying phrases.
• To overcome limitations of rule-based systems and distant supervision by creating a high-quality, manually-annotated dataset tailored to the syntactic and semantic complexity of scientific biology texts.

Proposed method

• The FOBIE dataset comprises 1,548 single sentences extracted from 1,292 unique full-text scientific biology documents, with no overlap between training (1,248), development (150), and test (150) sets.
• Each sentence is annotated with non-projective, n-ary relations between trigger words, argument phrases (central concepts), and modifying phrases, forming syntactically complex graphs.
• Keyphrases are not pre-classified into entity types, and the dataset focuses on relations such as trade-offs, correlations, and argument modifications, which are critical for abstracting biological problem spaces.
• A state-of-the-art neural relation extraction model, SciIE, is fine-tuned on FOBIE using span-based classification and joint learning of entities and relations, with ELMo embeddings and bi-LSTM representations.
• The model uses candidate span generation up to 14 tokens (to handle long keyphrases), attention mechanisms to identify syntactic heads, and joint optimization for span and relation prediction.
• A rule-based system (RBS) is also evaluated as a baseline to compare performance, demonstrating the superiority of neural methods on complex syntactic patterns.

Experimental results

Research questions

• RQ1 Can a manually-annotated, domain-independent relation extraction dataset improve the identification of trade-offs and related relations in scientific biology texts for engineers?
• RQ2 How effective are neural relation extraction models like SciIE in capturing complex, non-projective relations involving multi-word expressions and modifying phrases in long scientific sentences?
• RQ3 To what extent does syntactic variation in trade-off expressions hinder rule-based systems, and can neural models overcome this limitation?
• RQ4 How does the performance of relation extraction models vary across different relation types, particularly trade-offs versus argument modifiers or non-trade-off relations?

Key findings

• The neural model SciIE outperforms the rule-based system (RBS) significantly in detecting correct boundaries of trigger words and argument phrases, demonstrating the advantage of deep learning on complex syntactic patterns.
• SciIE achieves strong performance on trade-off relation extraction, with higher precision and recall on this relation type compared to others, despite the overall dataset being dominated by non-trade-off (54.05%) and argument-modifier (29.73%) relations.
• The model performs notably worse on argument-modifier relations due to the high syntactic variability in how arguments are modified, highlighting a key challenge in scientific IE.
• The dataset size (1,500 sentences) is comparable to existing scientific IE datasets, but FOBIE uniquely focuses on domain-independent relations critical for biomimetic design.
• The manual annotation process ensures high-quality, non-standardized keyphrases not found in standard knowledge bases, enabling training without reliance on distant supervision.
• FOBIE is publicly released with annotation guidelines to support reproducibility and further research in scientific information extraction for interdisciplinary engineering applications.