[Paper Review] Zap: Making Predictions Based on Online User Behavior

Yuri Chervonyi, Dragos Harabor+2 more|arXiv (Cornell University)|Jul 16, 2018

Data Visualization and Analytics8 references3 citations

TL;DR

Zap is a scalable, enterprise-ready machine learning pipeline that predicts online user behavior using sequential web interaction data. It enables website- and task-specific modeling with minimal website-specific code by using example generators, and achieves high predictive performance—often identifying 20% of users responsible for 80% of desired actions—through deep learning on real-time and archival data streams via Google Cloud and TensorFlow.

ABSTRACT

This paper introduces Zap, a generic machine learning pipeline for making predictions based on online user behavior. Zap combines well known techniques for processing sequential data with more obscure techniques such as Bloom filters, bucketing, and model calibration into an end-to-end solution. The pipeline creates website- and task-specific models without knowing anything about the structure of the website. It is designed to minimize the amount of website-specific code, which is realized by factoring all website-specific logic into example generators. New example generators can typically be written up in a few lines of code.

Motivation & Objective

• To address the lack of general-purpose, production-ready systems for predicting online user behavior using real-world web interaction data.
• To minimize website-specific logic in machine learning pipelines by abstracting it into simple example generators.
• To build a scalable, real-time prediction system that supports enterprise-level data workloads and integrates seamlessly with existing analytics stacks.
• To improve interpretability and trust in deep learning models for business applications through calibration curves and feature impact analysis.
• To enable non-expert teams to deploy high-impact, personalized user prediction models without deep machine learning expertise.

Proposed method

• Collects real-time user interaction events (e.g., clicks, scrolls, purchases) via a JavaScript library or Mobile SDK, streaming them to a server.
• Splits data into two streams: one stored in BigQuery for long-term archival and model training, and another in a low-latency database (e.g., MongoDB or Redis) for real-time inference.
• Constructs user sessions by grouping events by anonymousId and timestamp, filtering out bots, scrapers, and abnormally long sessions (e.g., >1000 events).
• Generates training examples via website-specific example generators—typically a few lines of code—that define features and labels from user sessions.
• Trains deep learning models using TensorFlow on Cloud Machine Learning Engine, leveraging sequential modeling techniques and techniques like Bloom filters and bucketing for efficiency.
• Serves real-time predictions using the low-latency database, with model inference exposed via an API for production use.

Experimental results

Research questions

• RQ1 How can a generic machine learning pipeline be designed to make accurate predictions on online user behavior without requiring deep website-specific logic?
• RQ2 What is the minimal amount of website-specific code required to build a production-grade user behavior prediction system?
• RQ3 Can a pipeline built on standard cloud infrastructure achieve low-latency real-time inference while maintaining high accuracy on sequential user interaction data?
• RQ4 How can model interpretability be improved in deep learning systems used for business decisions like retargeting or lead scoring?
• RQ5 To what extent can such a system identify high-propensity user segments (e.g., 20% of users driving 80% of conversions) in real-world applications?

Key findings

• Zap enables the creation of website- and task-specific prediction models using only a few lines of website-specific code in example generators, significantly reducing engineering overhead.
• The pipeline achieves a 20/80 performance pattern in production use cases, where 20% of users account for 80% of desired behaviors such as purchases or sign-ups.
• Calibration curves demonstrate that model confidence increases meaningfully for users with more than 10 clicks and over one minute of time-on-site, validating that key behavioral signals are captured.
• The model's positive prediction rate in the top 10% of predictions reaches 0.6 for high-engagement users, compared to 0.4 across all users, confirming the model's sensitivity to engagement signals.
• The system successfully decouples data collection, preprocessing, and model serving using Apache Beam and Google Cloud technologies, enabling scalable and reliable real-time inference.
• The approach supports interpretability through calibration analysis, allowing stakeholders to validate model behavior against business intuition without requiring full model retraining.