[Paper Review] Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

Jason Gauci, Edoardo Conti+8 more|arXiv (Cornell University)|Nov 1, 2018

Reinforcement Learning in Robotics42 references90 citations

TL;DR

Horizon is Facebook’s open source end-to-end platform for applied reinforcement learning on large real-world datasets, offering data preprocessing, feature normalization, data understanding, multi-node training, counterfactual policy evaluation, optimized serving, and production-grade deployments.

ABSTRACT

In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, optimized serving, and a model-based data understanding tool. We also showcase and describe real examples where reinforcement learning models trained with Horizon significantly outperformed and replaced supervised learning systems at Facebook.

Motivation & Objective

• Address the challenges of applying reinforcement learning to real, industry-produced data rather than simulated environments.
• Provide an end-to-end platform that supports data preprocessing, feature normalization, model training, evaluation, and serving for production RL use cases.
• Demonstrate real-world Facebook deployments where RL models trained with Horizon outperformed supervised or heuristic policies.
• Offer tools for problem formulation validation and data understanding to ensure RL is applied to well-posed tasks.

Proposed method

• End-to-end platform built in Python using PyTorch for modeling/training and Caffe2 for model serving.
• Spark-based Timeline data preprocessing pipeline to convert logged data into RL training format.
• Feature normalization workflow that automatically detects feature types and computes normalization parameters for training and serving.
• Data understanding tool that uses a probabilistic neural network with a Gaussian Mixture Model layer to assess MDP conformity and feature importance.
• Implementation of discrete-action DQN, parametric-action DQN, DDPG, and SAC to cover discrete, very large discrete, and continuous action spaces.
• Distributed training over CPU/GPU/multi-node environments with PyTorch multi-GPU support.
• Counterfactual policy evaluation (CPE) methods integrated into training and evaluation (DM, IS, DR, sequential DR, MAGIC).
• Optimized model serving by exporting PyTorch models to ONNX and then to Caffe2 for deployment at scale.

Experimental results

Research questions

• RQ1 How can RL be effectively applied to large-scale, real-world production data where simulators are unavailable or unsuitable?
• RQ2 What end-to-end tooling and data workflows are needed to preprocess, normalize, evaluate, and deploy RL models in production settings?
• RQ3 Do counterfactual policy evaluation methods provide reliable offline estimates to guide production RL without excessive online experimentation?
• RQ4 Can production RL models trained with Horizon demonstrate tangible improvements over supervised or heuristic policies in real applications?

Key findings

• Horizon supports training on millions of samples with multi-node and multi-GPU setups, enabling fast iteration on large feature spaces.
• Offline counterfactual policy evaluation methods are integrated into training, providing estimates of policy performance without immediate online deployment.
• Production deployments at Facebook showed RL models trained with Horizon outperforming supervised learning and heuristic policies for tasks like push notifications and page administrator notifications.
• A data understanding tool helps ensure RL problems conform to the MDP framework and identifies meaningful features and action-state relationships.
• Feature normalization and data preprocessing pipelines improve convergence and model performance on large, sparse, and heterogeneous feature sets.