[Paper Review] X-TIME: An in-memory engine for accelerating machine learning on tabular data with CAMs

Giacomo Pedretti, J. W. Moon+12 more|arXiv (Cornell University)|Apr 3, 2023

Network Packet Processing and Optimization4 citations

TL;DR

X-TIME proposes a novel in-memory computing architecture using high-precision analog content addressable memories (aCAMs) to accelerate inference of tree-based machine learning models like XGBoost and CatBoost. By mapping decision trees directly to analog CAM arrays for parallel, in-situ comparison of feature values against stored decision ranges, X-TIME achieves 9740× lower latency and 119× higher throughput than a V100 GPU at 19W power, with resilience to hardware variations through ensemble averaging.

ABSTRACT

Structured, or tabular, data is the most common format in data science. While deep learning models have proven formidable in learning from unstructured data such as images or speech, they are less accurate than simpler approaches when learning from tabular data. In contrast, modern tree-based Machine Learning (ML) models shine in extracting relevant information from structured data. An essential requirement in data science is to reduce model inference latency in cases where, for example, models are used in a closed loop with simulation to accelerate scientific discovery. However, the hardware acceleration community has mostly focused on deep neural networks and largely ignored other forms of machine learning. Previous work has described the use of an analog content addressable memory (CAM) component for efficiently mapping random forests. In this work, we develop an analog-digital architecture that implements a novel increased precision analog CAM and a programmable chip for inference of state-of-the-art tree-based ML models, such as XGBoost, CatBoost, and others. Thanks to hardware-aware training, X-TIME reaches state-of-the-art accuracy and 119x higher throughput at 9740x lower latency with >150x improved energy efficiency compared with a state-of-the-art GPU for models with up to 4096 trees and depth of 8, with a 19W peak power consumption.

Motivation & Objective

• Address the growing need for ultra-low latency inference in real-time applications using large tree-based ML models on tabular data.
• Overcome the performance bottlenecks of CPUs and GPUs in handling irregular memory access and thread synchronization in large tree ensembles.
• Design a holistic in-memory architecture that integrates high-precision analog CAMs with a reconfigurable on-chip network for scalable, energy-efficient inference.
• Enable support for diverse tree-based models (regression, multi-class classification) and inference techniques (e.g., input batching) on a single chip.
• Ensure robustness to hardware variations such as memristor conductance drift and DAC input errors through architectural resilience and error mitigation.

Proposed method

• Map decision trees to analog CAM arrays by encoding decision thresholds as analog ranges stored in memristor-based CAM cells.
• Implement a novel 8-bit precision analog CAM cell design to improve resolution and accuracy over prior 4-bit designs.
• Use in-core SRAM to store leaf values and perform local accumulation across multiple trees within a single core.
• Employ a reconfigurable H-tree network-on-chip (NoC) to interconnect 4096 cores, enabling scalable data flow and model mapping.
• Integrate in-network reduction logic to minimize data movement by accumulating leaf outputs as data propagate through the NoC.
• Apply defect-aware design principles, using experimental data from TaOx memristors and 16nm analog circuits to model and mitigate accuracy degradation from hardware variations.

Experimental results

Research questions

• RQ1 Can analog CAMs with increased precision (8-bit) significantly improve the accuracy and performance of tree-based ML inference compared to prior 4-bit designs?
• RQ2 How does the integration of in-memory computing with a reconfigurable NoC enable scalable, low-latency inference across diverse tree-based model topologies?
• RQ3 To what extent is the proposed architecture resilient to real-world hardware defects such as memristor conductance variation and DAC input error?
• RQ4 What performance gains can be achieved in latency and throughput for large-scale tree ensembles (e.g., 1M+ nodes) compared to state-of-the-art GPU and FPGA accelerators?
• RQ5 Can the architecture maintain high energy efficiency (e.g., low energy per decision) while supporting complex models and diverse inference workloads?

Key findings

• X-TIME achieves 9740× lower inference latency and 119× higher throughput than an NVIDIA V100 GPU for the Churn modeling dataset, with latency as low as ~100 ns.
• The architecture maintains constant latency and throughput regardless of the number of trees or leaves, depending only on feature count and number of classes, unlike GPU and FPGA which scale linearly with model size.
• The system consumes only 19W peak power, enabling ultra-low energy per decision of 0.3 nJ/decision, significantly outperforming GPU and SRAM-based accelerators.
• The model is resilient to hardware defects: a 0.2% memristor flip probability results in less than 0.5% accuracy drop, due to the ensemble nature of tree-based models.
• The 8-bit analog CAM design doubles precision over prior work, enabling higher accuracy without sacrificing area or power efficiency.
• The reconfigurable NoC and in-network reduction enable efficient support for diverse workloads, including input batching and multi-class classification, with minimal data movement.