[Paper Review] AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)

Phanwadee Sinthong, Michael J. Carey|arXiv (Cornell University)|Aug 19, 2019

Cloud Computing and Resource Management18 references4 citations

TL;DR

AFrame extends DataFrames to scale efficiently on large, semi-structured datasets by building on Apache AsterixDB, a distributed, managed big data system. It enables Pandas-like operations with transparent parallel execution, query optimization, and support for live data and ML models, outperforming JSON-based systems and matching Parquet-based ones while eliminating manual data management overhead.

ABSTRACT

Analyzing the increasingly large volumes of data that are available today, possibly including the application of custom machine learning models, requires the utilization of distributed frameworks. This can result in serious productivity issues for "normal" data scientists. This paper introduces AFrame, a new scalable data analysis package powered by a Big Data management system that extends the data scientists' familiar DataFrame operations to efficiently operate on managed data at scale. AFrame is implemented as a layer on top of Apache AsterixDB, transparently scaling out the execution of DataFrame operations and machine learning model invocation through a parallel, shared-nothing big data management system. AFrame incrementally constructs SQL++ queries and leverages AsterixDB's semistructured data management facilities, user-defined function support, and live data ingestion support. In order to evaluate the proposed approach, this paper also introduces an extensible micro-benchmark for use in evaluating DataFrame performance in both single-node and distributed settings via a collection of representative analytic operations. This paper presents the architecture of AFrame, describes the underlying capabilities of AsterixDB that efficiently support modern data analytic operations, and utilizes the proposed benchmark to evaluate and compare the performance and support for large-scale data analyses provided by alternative DataFrame libraries.

Motivation & Objective

• To address the productivity gap faced by data scientists when scaling from local to distributed data analysis.
• To provide a scale-independent DataFrame experience that maintains familiar Pandas-like semantics on large-scale, semi-structured data.
• To eliminate the need for data scientists to manage multiple systems (e.g., HDFS, Spark, TensorFlow) by integrating them into a single managed backend.
• To design a standardized, extensible micro-benchmark for evaluating DataFrame performance across single-node and distributed environments.
• To demonstrate that database-backed DataFrame systems can outperform traditional in-memory or file-based frameworks through query optimization and indexing.

Proposed method

• AFrame is implemented as a layer on top of Apache AsterixDB, translating DataFrame operations into SQL++ queries.
• It leverages AsterixDB’s support for semi-structured data, user-defined functions, and live data ingestion to enable scalable, managed analytics.
• The system uses lazy evaluation and database query optimization to minimize execution time and data movement.
• It supports direct invocation of pre-trained ML models (e.g., Scikit-Learn) without requiring distributed ML expertise.
• AFrame transparently scales operations across a shared-nothing, parallel architecture, enabling out-of-core processing on datasets larger than main memory.
• The proposed micro-benchmark isolates data preparation and expression execution times to provide fine-grained performance comparison across frameworks.

Experimental results

Research questions

• RQ1 Can a DataFrame system built on a full-featured big data management system achieve competitive performance on large-scale analytic workloads?
• RQ2 How does the performance of a managed, query-optimized DataFrame backend compare to in-memory or file-based DataFrame systems like Pandas or Spark?
• RQ3 To what extent do query optimization and indexing improve end-to-end time-to-insight in data analysis workflows?
• RQ4 Can a unified system reduce the complexity of integrating data ingestion, storage, query, and machine learning in data science pipelines?
• RQ5 How effective is the proposed micro-benchmark in revealing performance trade-offs across different DataFrame libraries and workloads?

Key findings

• AFrame outperforms JSON-based DataFrame systems by an order of magnitude on large datasets due to efficient query optimization and managed storage.
• The Parquet-based DataFrame in Spark performs competitively with AFrame due to columnar compression and efficient columnar operations.
• AFrame eliminates repetitive file scans by leveraging pre-ingested, indexed data in AsterixDB, reducing DataFrame creation overhead.
• AsterixDB’s query optimizer enables significant performance gains, especially in total evaluation time, which reflects real-world time-to-insight.
• The benchmark successfully isolates and quantifies performance differences, revealing that overheads in data preparation and expression execution vary significantly across systems.
• AFrame’s ability to natively support live data feeds and pre-trained ML models without additional engineering effort enhances productivity and usability.