[Paper Review] RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference

Mark Wilkening, Udit Gupta+5 more|arXiv (Cornell University)|Jan 29, 2021

Advanced Data Storage Technologies30 references4 citations

TL;DR

RecSSD proposes a near data processing (NDP) architecture that offloads embedding table operations—key bottlenecks in recommendation inference—directly to SSDs, reducing end-to-end inference latency by up to 2× compared to conventional SSDs. By executing gather and aggregation computations on the SSD itself and leveraging host- and device-side caching, RecSSD achieves up to 4× lower embedding operation latency on a real OpenSSD platform.

ABSTRACT

Neural personalized recommendation models are used across a wide variety of datacenter applications including search, social media, and entertainment. State-of-the-art models comprise large embedding tables that have billions of parameters requiring large memory capacities. Unfortunately, large and fast DRAM-based memories levy high infrastructure costs. Conventional SSD-based storage solutions offer an order of magnitude larger capacity, but have worse read latency and bandwidth, degrading inference performance. RecSSD is a near data processing based SSD memory system customized for neural recommendation inference that reduces end-to-end model inference latency by 2X compared to using COTS SSDs across eight industry-representative models.

Motivation & Objective

• Address the high infrastructure cost and performance bottleneck of large DRAM-based embedding tables in datacenter recommendation systems.
• Overcome the latency and bandwidth limitations of conventional SSDs when used for high-capacity, low-latency recommendation inference.
• Enable cost-effective, scalable deployment of large recommendation models by leveraging SSDs with intelligent on-device processing.
• Demonstrate practical feasibility of NDP in production-grade datacenter systems using open-source hardware and standard NVMe interfaces.

Proposed method

• Offloads entire embedding table operations (gather and aggregation) from the CPU to the SSD using near data processing (NDP) capabilities.
• Integrates custom firmware into the SSD’s FTL (Flash Translation Layer) to execute NDP workloads without changing host hardware or protocols.
• Employs a hybrid caching strategy combining host-side DRAM caching and SSD-side caching to exploit query locality patterns.
• Uses a modified UNVMe driver library to expose NDP interfaces to the host, enabling seamless integration with existing recommendation workloads.
• Leverages real-world trace data and synthetic workloads from Facebook’s DLRM to model production-scale access patterns.
• Implements the system on a real Cosmos+OpenSSD platform with Xilinx SDK and NVMe-compliant firmware, ensuring production readiness.

Figure 1. Recommendation models process both categorical and continuous input features.

Experimental results

Research questions

• RQ1 Can near data processing in SSDs significantly reduce end-to-end inference latency for neural recommendation models compared to conventional SSDs?
• RQ2 How effective is on-device computation of embedding operations in mitigating the performance gap between DRAM and SSD storage?
• RQ3 To what extent do host-side and SSD-side caching complement each other in reducing latency for recommendation inference workloads?
• RQ4 Can a fully software-implemented NDP solution be deployed on commodity SSDs without hardware modifications?
• RQ5 What is the performance impact of RecSSD across diverse, industry-representative recommendation models?

Key findings

• RecSSD reduces end-to-end inference latency by up to 2× across eight industry-representative recommendation models compared to baseline COTS SSDs.
• For memory-bound models dominated by embedding operations, RecSSD achieves up to 4× lower latency in individual embedding table operations.
• The system demonstrates comparable performance to DRAM-based systems for five of the eight models, which are compute-intensive and less memory-bound.
• The hybrid caching strategy of RecSSD reduces overall latency by effectively exploiting access locality patterns observed in real recommendation workloads.
• The implementation on a real OpenSSD platform with standard NVMe interfaces confirms the feasibility and practicality of deploying NDP in datacenter environments.
• The system achieves these gains without requiring hardware changes, relying solely on firmware and driver modifications within the FTL and UNVMe stack.

Figure 2. In order to support a high performance and simple logical block interface to the host while handling the peculiarities of NAND Flash memories, SSDs are designed with a Microprocessor operating alongside dedicated memory controllers.