[Paper Review] Towards Ontology-Enhanced Representation Learning for Large Language Models

Francesco Ronzano, Jay Nanavati|arXiv (Cornell University)|May 30, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a novel method to enhance embedding-based large language models (LLMs) by infusing knowledge from biomedical ontologies, specifically MONDO, using synthetic definition generation via GPT-3.5-turbo and contrastive fine-tuning. The approach improves in-domain sentence similarity performance without degrading out-of-domain capabilities, demonstrating consistent gains across multiple LLM architectures, especially for models pre-trained with basic methods.

ABSTRACT

Taking advantage of the widespread use of ontologies to organise and harmonize knowledge across several distinct domains, this paper proposes a novel approach to improve an embedding-Large Language Model (embedding-LLM) of interest by infusing the knowledge formalized by a reference ontology: ontological knowledge infusion aims at boosting the ability of the considered LLM to effectively model the knowledge domain described by the infused ontology. The linguistic information (i.e. concept synonyms and descriptions) and structural information (i.e. is-a relations) formalized by the ontology are utilized to compile a comprehensive set of concept definitions, with the assistance of a powerful generative LLM (i.e. GPT-3.5-turbo). These concept definitions are then employed to fine-tune the target embedding-LLM using a contrastive learning framework. To demonstrate and evaluate the proposed approach, we utilize the biomedical disease ontology MONDO. The results show that embedding-LLMs enhanced by ontological disease knowledge exhibit an improved capability to effectively evaluate the similarity of in-domain sentences from biomedical documents mentioning diseases, without compromising their out-of-domain performance.

Motivation & Objective

• To improve the representation learning of embedding-LLMs by integrating structured and linguistic knowledge from external ontologies.
• To address the limitation of domain-specific LLMs in capturing precise semantic relationships in specialized domains like biomedicine.
• To develop an automated, scalable method for knowledge infusion that leverages both ontology structure and generative LLMs.
• To evaluate the effectiveness of ontology-enhanced LLMs on in-domain and out-of-domain sentence similarity tasks.
• To ensure the method improves performance without compromising the model’s generalization ability.

Proposed method

• Generate synthetic, comprehensive concept definitions for all ontology concepts using GPT-3.5-turbo, leveraging linguistic (synonyms, descriptions) and structural (is-a relations) information from the ontology.
• Construct training samples for contrastive learning by pairing semantically similar and dissimilar definitions based on ontological relationships and semantic similarity.
• Fine-tune the target embedding-LLM using a contrastive learning objective that maximizes embedding similarity for positive pairs and minimizes it for negative pairs.
• Utilize a dual-encoder framework where the LLM encodes both positive and negative definition pairs during fine-tuning.
• Apply the knowledge infusion pipeline to MONDO, a biomedical disease ontology, to evaluate its impact on disease-related sentence similarity.
• Ensure reproducibility by open-sourcing the framework, prompts, hyperparameters, and evaluation procedures via a GitHub repository.

Figure 1: (a) structure of ontologies, generation of synthetic concept definitions; (b) creation of pairs os semantically related sentences by synonym substitution; (c) overview of the ontological knowledge infusion approach.

Experimental results

Research questions

• RQ1 Can injecting knowledge from a structured ontology like MONDO improve the sentence similarity performance of embedding-LLMs in the biomedical domain?
• RQ2 Does the proposed method maintain or improve the out-of-domain generalization capability of the LLMs after knowledge infusion?
• RQ3 How does the performance gain vary across different LLM architectures with varying pre-training strategies?
• RQ4 To what extent does using full-sentence definitions derived from ontological structure outperform methods based on noun phrases or synonyms alone?
• RQ5 Can a generative LLM effectively synthesize high-quality, diverse definitions that enhance contrastive fine-tuning?

Key findings

• Embedding-LLMs enhanced with ontological knowledge from MONDO show improved performance on in-domain sentence similarity tasks, particularly for models like PubMedBERT that used basic pre-training.
• The method maintains or slightly improves out-of-domain performance, indicating no degradation in generalization capabilities.
• GTEbase and GIST, which are more advanced models, show smaller but consistent improvements across most evaluation scenarios.
• SapBERT, a state-of-the-art biomedical LLM, achieves further performance gains after knowledge infusion, confirming the method’s effectiveness even on strong baselines.
• The approach demonstrates scalability and reproducibility, with full implementation and evaluation code publicly available on GitHub.
• The use of full-sentence definitions derived from ontological structure leads to more robust and semantically rich representations compared to synonym-based methods.