[Paper Review] DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population

Ningyu Zhang, Xin Xu+20 more|arXiv (Cornell University)|Jan 10, 2022

Topic Modeling5 citations

TL;DR

DeepKE is an open-source, extensible deep learning toolkit for knowledge base population that supports named entity recognition, relation extraction, and attribute extraction across low-resource, document-level, and multimodal scenarios. It provides a unified, modular framework with pre-trained models, automatic hyperparameter tuning, and real-time inference, achieving state-of-the-art performance in few-shot and multimodal settings compared to existing baselines.

ABSTRACT

We present an open-source and extensible knowledge extraction toolkit DeepKE, supporting complicated low-resource, document-level and multimodal scenarios in the knowledge base population. DeepKE implements various information extraction tasks, including named entity recognition, relation extraction and attribute extraction. With a unified framework, DeepKE allows developers and researchers to customize datasets and models to extract information from unstructured data according to their requirements. Specifically, DeepKE not only provides various functional modules and model implementation for different tasks and scenarios but also organizes all components by consistent frameworks to maintain sufficient modularity and extensibility. We release the source code at GitHub in https://github.com/zjunlp/DeepKE with Google Colab tutorials and comprehensive documents for beginners. Besides, we present an online system in http://deepke.openkg.cn/EN/re_doc_show.html for real-time extraction of various tasks, and a demo video.

Motivation & Objective

• To address the limitations of existing IE toolkits that support only single tasks and struggle in complex, real-world scenarios such as low-resource, document-level, and multimodal settings.
• To develop a unified, extensible framework that enables researchers and developers to customize datasets and models for diverse knowledge extraction tasks without extensive low-level coding.
• To improve performance in low-resource scenarios through few-shot learning methods like KnowPrompt and prototype-based NER, and in multimodal settings via vision-language integration.
• To provide comprehensive tooling, including Google Colab tutorials, documentation, and an online real-time extraction system, to lower the barrier to entry for new users.
• To ensure long-term maintainability with active support, bug fixes, and feature extensions based on community feedback.

Proposed method

• DeepKE employs a unified framework for data processing, model training, and evaluation, enabling seamless integration of various NLP tasks and scenarios.
• It leverages pre-trained language models (e.g., BERT, RoBERTa) as encoders for sequence-level encoding in NER, RE, and attribute extraction tasks.
• For few-shot learning, it implements KnowPrompt—a prompt-tuning method with knowledge injection—and compares against prototype-based and example-based baselines.
• In document-level relation extraction, it uses context-aware encoders (e.g., BERT, RoBERTa) and supports intra- and inter-sentence relation reasoning on large-scale datasets like DocRED.
• For multimodal extraction, it integrates vision and text encoders (e.g., ViLBERT, UMT) to jointly process images and text, improving performance on Twitter-2017 and MNRE datasets.
• The toolkit supports automatic hyperparameter tuning and modular design, allowing users to plug in custom models and datasets with minimal code.

Experimental results

Research questions

• RQ1 Can a unified deep learning toolkit effectively support multiple knowledge extraction tasks—named entity recognition, relation extraction, and attribute extraction—across diverse and complex real-world scenarios?
• RQ2 How does the performance of DeepKE compare to state-of-the-art methods in low-resource few-shot settings for both NER and relation extraction?
• RQ3 To what extent can DeepKE capture long-range dependencies and inter-sentence relations in document-level knowledge extraction tasks?
• RQ4 Can multimodal integration of text and image inputs improve the accuracy of named entity recognition and relation extraction compared to unimodal approaches?
• RQ5 How effective is the unified framework in enabling rapid customization, hyperparameter tuning, and deployment for researchers and developers?

Key findings

• In few-shot NER, DeepKE with KnowPrompt achieved an F1 score of 74.3 on the 8-shot setting, outperforming baselines such as PTR (70.5) and GDPNet (42.0).
• For few-shot relation extraction on SemEval-2010 Task-8, DeepKE (KnowPrompt) achieved 84.8 F1 in the 32-shot setting, surpassing PTR (84.2) and GDPNet (81.2).
• In document-level relation extraction on DocRED, DeepKE with RoBERTa-large achieved performance comparable to or better than SOTA models like Coref and ATLOP.
• In multimodal NER on Twitter-2017, DeepKE demonstrated improved performance over baselines such as AdapCoAtt-BERT-CRF, ViLBERT, and UMT.
• In multimodal relation extraction on MNRE, DeepKE outperformed BERT+SG, BERT+SG+Att, and MEGA, showing strong generalization in vision-language reasoning.
• The online system hosted at http://deepke.zjukg.cn/ enables real-time extraction using pre-defined schemas without model fine-tuning, enhancing usability for end-users.