[Paper Review] On Leveraging Large Language Models for Enhancing Entity Resolution: A Cost-efficient Approach

Huahang Li, Longyu Feng+4 more|arXiv (Cornell University)|Jan 7, 2024

Data Quality and Management6 citations

TL;DR

The paper proposes an LLM-based, cost-aware framework for entity resolution that selects an optimal set of matching questions (MQsSP) to reduce uncertainty (entropy) within a given budget, using LLM confidence to adjust partition probabilities.

ABSTRACT

Entity resolution, the task of identifying and merging records that refer to the same real-world entity, is crucial in sectors like e-commerce, healthcare, and law enforcement. Large Language Models (LLMs) introduce an innovative approach to this task, capitalizing on their advanced linguistic capabilities and a ``pay-as-you-go'' model that provides significant advantages to those without extensive data science expertise. However, current LLMs are costly due to per-API request billing. Existing methods often either lack quality or become prohibitively expensive at scale. To address these problems, we propose an uncertainty reduction framework using LLMs to improve entity resolution results. We first initialize possible partitions of the entity cluster, refer to the same entity, and define the uncertainty of the result. Then, we reduce the uncertainty by selecting a few valuable matching questions for LLM verification. Upon receiving the answers, we update the probability distribution of the possible partitions. To further reduce costs, we design an efficient algorithm to judiciously select the most valuable matching pairs to query. Additionally, we create error-tolerant techniques to handle LLM mistakes and a dynamic adjustment method to reach truly correct partitions. Experimental results show that our method is efficient and effective, offering promising applications in real-world tasks.

Motivation & Objective

• Motivate the use of Large Language Models (LLMs) for improving entity resolution in data-rich domains.
• Introduce a cost-aware workflow that balances accuracy and LLM usage cost.
• Define and solve the Matching Questions Selection Problem (MQsSP) to minimize uncertainty under budget constraints.
• Model how LLM responses adjust partition probabilities and reduce entropy in the result set.

Proposed method

• Represent records as nodes in a graph and define possible partitions of records into clusters with associated probabilities.
• Use Shannon entropy to quantify uncertainty in the result set of partitions (RS).
• Define Matching Questions (MQ) and a cost function F(MQ) to capture LLM pricing, enabling budget-constrained MQ selection.
• Formulate MQsSP as maximizing joint entropy reduction under a knapsack-like budget and propose a greedy, sub-modular optimization approach with a (1-1/e) guarantee.
• Develop an LLM-assisted ER workflow (Algorithm 2) that iteratively selects MQs, queries the LLM, and updates partition probabilities.
• Provide a probabilistic adjustment model (including Cap and Conf) to account for imperfect LLM responses when updating RS.

Figure 1 : An illustration of possible matches (record linkages) in Table II .The probability associated with each linkage is the cumulative sum of its occurrences across possible partitions.

Experimental results

Research questions

• RQ1 How can LLMs be used as a service to improve entity resolution while controlling cost?
• RQ2 What is an effective method to select a set of matching questions that maximally reduce uncertainty under a budget?
• RQ3 How can LLM responses be integrated to refine the probability distribution over possible partitions?
• RQ4 What is the computational hardness of MQsSP and can it be efficiently approximated in practice?

Key findings

• The MQsSP is NP-hard, even in a special case with k=1 (reducible to 0/1 Knapsack).
• A greedy approximation using sub-modularity yields an effective budget-constrained MQ selection strategy with a (1-1/e) performance guarantee.
• Entropy-based framing allows quantifying uncertainty reduction when incorporating LLM responses.
• The framework demonstrates cost-effective reduction of uncertainty in entity resolution by adjusting partition distributions based on LLM answers and confidence.
• Experiments on datasets (ACM, Amazon-eBay, Electronics) show the approach can reduce uncertainty under budget while using an established ER tool for initial partitioning.

Figure 2 : A depiction of our proposed approach’s workflow: In this example, ChatGPT can generate responses to determine if they pertain to the same records. With a specified confidence threshold, we can then refine the probability distribution of Possible Partitions using the Naive Bayes.