[Paper Review] The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox

Daniel Crankshaw, Peter Bailis+6 more|arXiv (Cornell University)|Sep 12, 2014

Scientific Computing and Data Management22 references84 citations

TL;DR

Velox is a scalable, low-latency model management and serving system designed to bridge the gap between offline model training and real-time inference in large-scale analytics pipelines. By integrating with Spark and Tachyon, Velox enables efficient model serving, dynamic model updates, and adaptive maintenance strategies, achieving prediction accuracy close to batch retraining while reducing latency for production workloads.

ABSTRACT

To support complex data-intensive applications such as personalized recommendations, targeted advertising, and intelligent services, the data management community has focused heavily on the design of systems to support training complex models on large datasets. Unfortunately, the design of these systems largely ignores a critical component of the overall analytics process: the deployment and serving of models at scale. In this work, we present Velox, a new component of the Berkeley Data Analytics Stack. Velox is a data management system for facilitating the next steps in real-world, large-scale analytics pipelines: online model management, maintenance, and serving. Velox provides end-user applications and services with a low-latency, intuitive interface to models, transforming the raw statistical models currently trained using existing offline large-scale compute frameworks into full-blown, end-to-end data products capable of recommending products, targeting advertisements, and personalizing web content. To provide up-to-date results for these complex models, Velox also facilitates lightweight online model maintenance and selection (i.e., dynamic weighting). In this paper, we describe the challenges and architectural considerations required to achieve this functionality, including the abilities to span online and offline systems, to adaptively adjust model materialization strategies, and to exploit inherent statistical properties such as model error tolerance, all while operating at "Big Data" scale.

Motivation & Objective

• To address the missing piece in complex analytics pipelines: scalable, low-latency model serving and management after offline training.
• To enable end-to-end data products by transforming raw statistical models into production-ready services with minimal latency.
• To support both offline retraining and online incremental model maintenance using adaptive strategies.
• To reduce operational complexity by abstracting model lifecycle management from application logic.
• To leverage semantic knowledge of models (e.g., error tolerance, materialization) for efficient caching and replication across clusters.

Proposed method

• Velox exposes models via a generic RESTful API for low-latency prediction across diverse query types.
• It integrates with existing cluster frameworks like Spark and Tachyon for offline training and distributed data storage.
• Velox employs adaptive model materialization strategies that balance freshness, latency, and resource cost.
• It supports online model maintenance using bandit-based learning and dynamic model weighting to adapt to changing user patterns.
• The system exploits statistical properties such as model error tolerance to optimize caching and replication.
• Velox uses a declarative interface to describe models as Spark User-Defined Functions (UDFs), enabling automated service exposure and lifecycle management.

Experimental results

Research questions

• RQ1 How can model serving and management be decoupled from offline training systems to enable low-latency production use?
• RQ2 What architectural patterns enable scalable, real-time model serving while maintaining accuracy close to batch retraining?
• RQ3 How can online incremental updates and dynamic model selection improve responsiveness to evolving data patterns?
• RQ4 What role do semantic properties of models (e.g., error tolerance) play in optimizing caching and replication strategies?
• RQ5 How can existing cluster compute frameworks be leveraged to automate model maintenance without sacrificing performance?

Key findings

• Velox’s prototype demonstrated strong performance on the MovieLens10M dataset, achieving low-latency predictions suitable for real-time use.
• The online incremental update strategy closely recovered the prediction accuracy of full offline batch retraining, validating its effectiveness.
• Adaptive model materialization strategies reduced latency and resource overhead by intelligently caching and replicating models based on access patterns and error tolerance.
• Integration with Spark and Tachyon enabled seamless deployment and scaling of model serving across large clusters.
• The system successfully abstracted model lifecycle management from application code, simplifying deployment of data products.
• Velox’s approach reduced the need for custom model-serving logic in production systems, lowering engineering and operational overhead.