[Paper Review] Dataset-On-Demand: Automatic View Search and Presentation for Data Discovery

Raul Castro Fernandez, Nan Tang+3 more|arXiv (Cornell University)|Nov 26, 2019

Data Quality and Management28 references4 citations

TL;DR

This paper introduces Dataset-On-Demand (DoD), a system that automates the discovery of semantically correct data views by allowing users to specify a desired schema, then automatically searching across heterogeneous data sources. Using a novel 4C classification method—compatible, contained, complementary, and contradictory—DoD reduces the number of candidate views by 2–10x, enabling users to efficiently select the correct view from a significantly pruned set within minutes.

ABSTRACT

Many data problems are solved when the right view of a combination of datasets is identified. Finding such a view is challenging because of the many tables spread across many databases, data lakes, and cloud storage in modern organizations. Finding relevant tables, and identifying how to combine them is a difficult and time-consuming process that hampers users' productivity. In this paper, we describe Dataset-On-Demand (DoD), a system that lets users specify the schema of the view they want, and have the system find views for them. With many underlying data sources, the number of resulting views for any given query is high, and the burden of choosing the right one is onerous to users. DoD uses a new method, 4C, to reduce the size of the view choice space for users. 4C classifies views into 4 classes: compatible views are exactly the same, contained views present a subsumption relationship, complementary views are unionable, and contradictory views have incompatible values that indicate fundamental differences between views. These 4 classes permit different presentation strategies to reduce the total number of views a user must consider. We evaluate DoD on two key metrics of interest: its ability to reduce the size of the choice space, and the end to end performance. DoD finds all views within minutes, and reduces the number of views presented to users by 2-10x.

Motivation & Objective

• Address the challenge of finding correct data views across large, heterogeneous data sources in modern organizations.
• Reduce the cognitive and time burden on users when selecting the right combination of tables and joins for a desired view.
• Overcome limitations of existing query-by-example systems that assume a PK-FK graph and cannot handle semantic ambiguity or erroneous joins.
• Enable users to specify a desired view via attributes and sample tuples, without requiring schema knowledge or manual join path definition.
• Minimize user involvement to the final selection phase by intelligently pruning and classifying candidate views using semantic and structural analysis.

Proposed method

• Accepts user input in the form of a query view specifying desired attributes and sample tuples to define the target schema.
• Automatically discovers all candidate views that satisfy the query by exploring all possible table combinations and join paths.
• Applies the 4C classification method to group views into four semantic categories: compatible (identical), contained (subsumption), complementary (unionable), and contradictory (incompatible values).
• Uses a 4C-chasing algorithm to detect and group views into these categories based on attribute semantics, value overlap, and structural consistency.
• Employs a presentation strategy that combines compatible views, prunes contained views, and highlights complementary or contradictory views to reduce user cognitive load.
• Optimizes view search performance using sampling joins and query pushdown techniques to manage memory and execution time on large tables.

Experimental results

Research questions

• RQ1 How can the number of candidate views be reduced when users specify a desired data view across heterogeneous, distributed data sources?
• RQ2 What semantic and structural criteria can effectively classify candidate views to support intuitive user selection?
• RQ3 Can a system reduce user involvement in view discovery while maintaining accuracy and performance in real-world data environments?
• RQ4 How effective is the 4C classification method in distinguishing semantically different views, such as work vs. home addresses, in the presence of ambiguous or erroneous joins?
• RQ5 To what extent can automated view search and classification outperform traditional manual or heuristic-based data discovery approaches?

Key findings

• DoD reduces the size of the view choice space by a factor of 2 to 10x through 4C classification, significantly decreasing user cognitive load.
• The system finds all candidate views within minutes—often in under 10 seconds for smaller queries—demonstrating strong end-to-end performance.
• The 4C method effectively identifies and groups views based on semantic similarity, subsumption, unionability, and incompatibility, enabling meaningful presentation strategies.
• Optimizations including sampling joins and query pushdown reduce end-to-end execution time by 4–6x compared to a baseline implementation.
• The system handles semantic heterogeneity and erroneous joins without assuming a PK-FK graph, making it robust in real-world environments with inconsistent or ambiguous schema mappings.
• The multi-row presentation strategy further reduces user involvement by summarizing views, allowing users to make final selections from a compact, semantically grouped set.