[Paper Review] METAM: Goal-Oriented Data Discovery

Sainyam Galhotra, Yue Gong+1 more|arXiv (Cornell University)|Apr 18, 2023

Data Quality and ManagementDecision Sciences3 citations

TL;DR

Metam introduces a goal-oriented data discovery framework that uses interventional queries to automatically identify and prioritize useful dataset augmentations by leveraging data properties, utility function monotonicity, and solution set sparsity. It achieves high utility gains with minimal queries, outperforming baselines on diverse ML and causal inference tasks.

ABSTRACT

Data is a central component of machine learning and causal inference tasks. The availability of large amounts of data from sources such as open data repositories, data lakes and data marketplaces creates an opportunity to augment data and boost those tasks' performance. However, augmentation techniques rely on a user manually discovering and shortlisting useful candidate augmentations. Existing solutions do not leverage the synergy between discovery and augmentation, thus under exploiting data. In this paper, we introduce METAM, a novel goal-oriented framework that queries the downstream task with a candidate dataset, forming a feedback loop that automatically steers the discovery and augmentation process. To select candidates efficiently, METAM leverages properties of the: i) data, ii) utility function, and iii) solution set size. We show METAM's theoretical guarantees and demonstrate those empirically on a broad set of tasks. All in all, we demonstrate the promise of goal-oriented data discovery to modern data science applications.

Motivation & Objective

• To address the limitation of traditional data discovery systems that fail to identify task-relevant augmentations due to lack of goal-awareness.
• To close the gap between data discovery and data augmentation by creating a feedback loop where the downstream task guides discovery.
• To reduce manual effort in dataset augmentation by automating the selection of useful joinable datasets without requiring users to specify criteria in advance.
• To optimize query efficiency in large-scale data repositories by exploiting structural properties of data, utility functions, and sparse solution sets.

Proposed method

• Metam uses data profiles—such as semantic similarity, correlation, and mutual information—to cluster augmentations and reduce redundant exploration.
• It applies Thompson sampling to rank clusters based on expected utility gain, prioritizing those most likely to improve downstream task performance.
• The framework enforces monotonicity in utility functions by wrapping the task implementation to discard augmentations that reduce performance.
• It employs combinatorial testing to prioritize small subsets of join paths over larger ones, exploiting the sparsity of useful augmentations.
• An anytime algorithm incrementally improves the solution by adaptively querying the most promising candidates based on cluster quality and utility estimates.
• The method is designed to be agnostic to the specific downstream task, requiring only a utility function to evaluate performance.

Experimental results

Research questions

• RQ1 Can a feedback loop between the downstream task and data discovery improve the identification of useful dataset augmentations compared to traditional discover-then-augment pipelines?
• RQ2 How can data properties such as similarity in data profiles be leveraged to reduce the number of candidate queries in large-scale data repositories?
• RQ3 To what extent does enforcing monotonicity in the utility function improve the efficiency and robustness of the discovery process?
• RQ4 Can combinatorial testing on join paths effectively prioritize sparse, high-utility augmentation sets without exhaustive enumeration?
• RQ5 How does the integration of clustering, ranking, and adaptive querying impact query efficiency and solution quality in real-world data discovery tasks?

Key findings

• Metam identifies useful augmentations among millions of candidates within minutes, demonstrating scalability and efficiency.
• The framework outperforms all ablation variants—Eq, Nc, and NcEq—by effectively combining clustering and Thompson sampling to avoid redundant queries.
• Varying the clustering hyperparameter ε has minimal impact on query count, indicating robustness to cluster granularity changes.
• Metam achieves high utility gains on diverse tasks, including classification, regression, causal inference (what-if/how-to analysis), entity linking, and clustering.
• The ablation study confirms that ignoring clustering (Nc) or uniform ranking (Eq) leads to significantly worse performance, validating the synergy of the proposed components.
• The method is effective even for non-obvious augmentations—such as taxi trip counts or grocery store presence—correlated with housing prices, which domain experts might overlook.