[Paper Review] Towards Open-Text Semantic Parsing via Multi-Task Learning of Structured Embeddings

Antoine Bordes, Xavier Glorot+2 more|arXiv (Cornell University)|Jul 19, 2011

Natural Language Processing Techniques33 references4 citations

TL;DR

This paper proposes a multi-task learning framework that uses structured embeddings to perform open-domain semantic parsing by jointly training on WordNet, ConceptNet, and raw text, enabling disambiguated meaning representation (MR) inference from free text with minimal direct supervision. The model achieves strong performance in word-sense disambiguation and MR ranking, with median ranks of 516/41,024 for predicted MRs and 640/41,024 for correct ones, demonstrating effective integration of lexical, commonsense, and syntactic knowledge.

ABSTRACT

Open-text (or open-domain) semantic parsers are designed to interpret any statement in natural language by inferring a corresponding meaning representation (MR). Unfortunately, large scale systems cannot be easily machine-learned due to lack of directly supervised data. We propose here a method that learns to assign MRs to a wide range of text (using a dictionary of more than 70,000 words, which are mapped to more than 40,000 entities) thanks to a training scheme that combines learning from WordNet and ConceptNet with learning from raw text. The model learns structured embeddings of words, entities and MRs via a multi-task training process operating on these diverse sources of data that integrates all the learnt knowledge into a single system. This work ends up combining methods for knowledge acquisition, semantic parsing, and word-sense disambiguation. Experiments on various tasks indicate that our approach is indeed successful and can form a basis for future more sophisticated systems.

Motivation & Objective

• To enable open-domain semantic parsing by learning meaning representations (MRs) from raw text without requiring large-scale manually annotated data.
• To address the challenge of word-sense disambiguation (WSD) at scale by leveraging structured embeddings trained on diverse knowledge sources.
• To unify representations of MRs, WordNet synsets, and ConceptNet relations in a single vector space to enable joint learning and inference.
• To enrich existing knowledge bases like WordNet by discovering new, contextually grounded relations through raw text mining.
• To develop a scalable, end-to-end system capable of assigning disambiguated MRs to any natural language sentence using indirect supervision.

Proposed method

• The model uses a structured embedding framework to represent words, entities (synsets), and meaning representations (MRs) as low-dimensional vectors in a shared space.
• It employs an energy-based scoring function to evaluate the plausibility of MRs, where lower energy indicates higher likelihood.
• Multi-task training is performed across three data sources: WordNet (for sense definitions and relations), ConceptNet (for commonsense relations), and raw Wikipedia text (for contextual usage).
• The system performs two-stage inference: first, semantic role labeling (SRL) identifies the MR structure; second, word-sense disambiguation selects the correct synset for each lemma using the energy function.
• The model learns to transfer knowledge from WordNet and ConceptNet to raw text by jointly optimizing the energy function across all data sources.
• It enables knowledge enrichment by identifying novel, contextually valid relations between lemmas and synsets not present in WordNet or ConceptNet.

Experimental results

Research questions

• RQ1 Can a single structured embedding model effectively learn to infer disambiguated meaning representations from raw text using only indirect supervision?
• RQ2 To what extent can multi-task learning across WordNet, ConceptNet, and raw text improve word-sense disambiguation and MR prediction in open-domain settings?
• RQ3 How well can the model generalize to unseen word-sense combinations and produce plausible MRs without explicit training on them?
• RQ4 Can the model discover and encode new, non-trivial relations between entities that are not present in existing knowledge bases?
• RQ5 How does the energy-based scoring function compare to existing methods like WordNet::Similarity in ranking correct MRs?

Key findings

• The model achieves a median rank of 516 out of 41,024 for predicted meaning representations on the XWN test set, indicating strong generalization and plausibility scoring.
• Correct MRs are ranked with a median of 640/41,024, showing the model effectively learns to assign lower energy to semantically valid MRs.
• The model significantly outperforms WordNet::Similarity, which achieves a median rank of 13,500/41,024, due to its ability to learn richer, context-sensitive relations beyond WordNet’s limited relation types.
• The system successfully discovers and reports novel relation types not present in WordNet or ConceptNet, such as 'earn' linking 'students' to 'money', demonstrating knowledge enrichment potential.
• Compared to TextRunner, the model provides disambiguated entity links and integrates with existing knowledge bases, while TextRunner lacks sense disambiguation and grounding to external resources.
• The multi-task learning setup enables effective transfer of knowledge from structured knowledge bases to raw text, resulting in robust and scalable semantic parsing with minimal direct supervision.