[Paper Review] DeepType: Multilingual Entity Linking by Neural Type System Evolution

Jonathan Raiman, Olivier Raiman|arXiv (Cornell University)|Feb 3, 2018

Topic Modeling72 citations

TL;DR

DeepType jointly designs a neural type system from an ontology and uses it to constrain a neural entity linking model, enabling near-human disambiguation accuracy and cross-lingual transfer without retraining for new entities.

ABSTRACT

The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data. DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system. First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters. We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

Motivation & Objective

• Integrate symbolic knowledge into neural reasoning via an automatically designed type system for entity linking (EL).
• Reduce EL disambiguation complexity and improve accuracy by constraining outputs with type axes derived from ontologies.
• Demonstrate multilingual and cross-lingual transfer of the learned representations and assess the impact on NER transfer.
• Compare machine-designed type systems to human-designed baselines across standard EL datasets (WikiDisamb30, CoNLL-YAGO, TAC KBP 2010).
• Evaluate whether DeepType pretraining aids downstream NER tasks and whether bilingual training benefits multilingual performance.

Proposed method

• Reformulate type system design as a mixed-integer problem where discrete variables select ontology-derived type axes and continuous variables fit a classifier to the type system.
• Use a two-step optimization: (i) discrete optimization of the type system with an Oracle and a Learnability heuristic to estimate learnability and disambiguation power, (ii) gradient descent to train the type classifier and entity predictor.
• Define a Type Axis as a root-edge pair over an ontology, and constrain entity predictions to respect type memberships via a probabilistic scoring formula.
• Compute an objective J(A) combining Oracle disambiguation power, Learnability, and a penalty for larger type system size to guide axis selection.
• Train a Type Classifier (bi-directional LSTM) over multilingual text to predict type labels for tokens, enabling cross-lingual supervision.
• At inference, combine type-based beliefs with a base entity-link score to rank candidate entities, using a soft combination of the type probabilities and a LinkCount baseline.

Experimental results

Research questions

• RQ1 Can an automatically designed neural type system improve entity linking beyond human-designed type systems?
• RQ2 Do machine-discovered type systems generalize across languages (English, French, German, Spanish) for EL?
• RQ3 Does incorporating symbolic type constraints reduce disambiguation complexity from O(N^2) to O(N) and enable adding new entities without retraining?
• RQ4 Can DeepType’s type-informed representations transfer to NER and other downstream tasks?
• RQ5 How do different search methods (Beam, Greedy, GA, CEM) compare in discovering effective type systems?

Key findings

• DeepType outperforms previous EL methods on WikiDisamb30, CoNLL (YAGO), and TAC KBP 2010 across multilingual setups.
• Oracle-based disambiguation reaches 99.0% on CoNLL (YAGO) and 98.6% on TAC KBP 2010, suggesting near-solution potential with improved type prediction.
• Machines-designed type systems outperform human-designed systems on WikiDisamb30, CoNLL (YAGO), and TAC KBP 2010 across several search methods.
• Type systems learned with bilingual English–French training generalize to French, German, and Spanish without degradation, and bilingual training can be beneficial.
• Pre-training for NER with DeepType yields improved F1 and even state-of-the-art on OntoNotes dev, showing cross-domain transfer benefits.
• The approach reduces EL complexity to O(N) and enables integrating new entities without retraining.