[Paper Review] DataHub: Collaborative Data Science & Dataset Version Management at Scale

Anant Bhardwaj, Souvik Bhattacherjee+5 more|arXiv (Cornell University)|Sep 2, 2014

Scientific Computing and Data Management4 citations

TL;DR

This paper proposes DataHub, a collaborative data science platform with a dataset version control system (DSVC) that enables scalable, versioned management of large, evolving datasets. Inspired by Git, DSVC supports branching, merging, and querying across datasets using a dual representation (version-first and record-first) to balance storage efficiency and query performance, significantly reducing redundancy and improving reproducibility in data science teams.

ABSTRACT

Relational databases have limited support for data collaboration, where teams collaboratively curate and analyze large datasets. Inspired by software version control systems like git, we propose (a) a dataset version control system, giving users the ability to create, branch, merge, difference and search large, divergent collections of datasets, and (b) a platform, DataHub, that gives users the ability to perform collaborative data analysis building on this version control system. We outline the challenges in providing dataset version control at scale.

Motivation & Objective

• To address the lack of scalable, collaborative dataset versioning in data science workflows, especially for large, heterogeneous, and evolving datasets.
• To reduce storage bloat and cost in research teams by enabling efficient version management and eliminating redundant data copies.
• To provide a system that supports provenance tracking, version querying, and collaboration across multiple users and datasets.
• To design a dual-representation storage model (version-first and record-first) that optimizes both retrieval efficiency and query performance.
• To build a hosted platform (DataHub) that simplifies collaborative data analysis with tools for cleaning, search, and visualization, analogous to GitHub for code.

Proposed method

• Designing a dataset version control system (DSVC) inspired by Git but extended for structured, large-scale datasets with rich query capabilities.
• Implementing a dual storage representation: version-first for efficient version retrieval and record-first for efficient property-based version discovery.
• Using compressed bitmaps to represent version membership of records in branching version graphs, enabling efficient set operations for queries.
• Employing optimization techniques to select optimal encodings of version graphs, including the use of 'Steiner datasets' to reduce retrieval costs.
• Integrating indexing and caching of frequently executed VQL (Version Query Language) queries to improve performance.
• Building a hosted platform (DataHub) on top of DSVC with tools for data cleaning, search, integration, and visualization to support collaborative data science.

Experimental results

Research questions

• RQ1 How can dataset versioning be scaled to support large, heterogeneous, and rapidly evolving datasets in collaborative data science environments?
• RQ2 What dual data representation strategies can balance storage efficiency and query performance in multi-version dataset management?
• RQ3 How can versioning systems reduce storage costs and eliminate redundant data copies in large-scale research teams?
• RQ4 What mechanisms enable efficient provenance tracking and reproducibility of data analyses across multiple users and versions?
• RQ5 How can a hosted platform enhance collaboration in data science by integrating versioning with data cleaning, search, and visualization tools?

Key findings

• The dual representation model (version-first and record-first) enables efficient retrieval of complete versions and efficient discovery of versions satisfying specific conditions, balancing performance and storage.
• Using compressed bitmaps to represent record version membership allows efficient set operations (e.g., OR, AND) for complex version queries, improving query efficiency.
• The system reduces storage costs by eliminating redundant data copies, with potential savings of up to $100K/year in large research teams, as demonstrated in a case study of a computational biology group.
• The use of Steiner datasets—additional intermediate versions—can reduce retrieval costs by optimizing the version graph structure, though the search space for optimal encoding is large and requires heuristics.
• The platform design enables backward compatibility with existing file-level APIs, allowing researchers to use existing scripts without modification, reducing adoption barriers.
• The system supports provenance tracking by enabling users to determine who last used a dataset and whether it is essential for reproducibility, reducing stress from data deletion.