[Paper Review] NCAM: Near-Data Processing for Nearest Neighbor Search

Vincent T. Lee, Carlo C. Del Mundo+4 more|arXiv (Cornell University)|Jun 12, 2016

Network Packet Processing and Optimization4 citations

TL;DR

This paper proposes NCAM, a near-data processing architecture using 3D-stacked memory to perform exact k-nearest neighbor (kNN) search in-place, eliminating costly data movement. By leveraging processing-in-memory (PIM) and emerging memory technologies, NCAM achieves up to 160x higher throughput per watt and three orders of magnitude lower latency than a NVIDIA Titan X GPU for server workloads.

ABSTRACT

Deep down in many applications like natural language processing (NLP), vision, and robotics is a form of the k-nearest neighbor search algorithm (kNN). The kNN algorithm is primarily bottlenecked by data movement, limiting throughput and incurring latency in these applications. While there do exist well bounded kNN approximations that improve the performance of kNN, these algorithms trade-off accuracy and quickly degrade into linear search for high dimensionality. To address data movement, we designed the nearest neighbor content addressable memory (NCAM) which employs processing in-memory (PIM) to eliminate costly data transfers, and provides exact nearest neighbor search. NCAMs benefit from the modularity offered by 3D die-stacking technology which allows us to side-step the issues of direct integration with DRAM dies. The NCAM benefits from the higher density and speeds of emerging memory technologies and interfaces. We characterize a state-of-the-art software kNN implementation and expose the shortcomings of approximate kNN search. We present a full NCAM design and estimate its performance using post-placement and route estimates, and we show that its power characteristics are compatible with emerging memory substrates. We then evaluate energy efficiency and latency compared to modern multi-core CPUs and GPGPU platforms using parameters typical of mobile and server workloads. Our simulation results show that the NCAM can achieve up to 160x throughput per watt and three orders of magnitude latency improvement over a NVIDIA Titan X GPU for server workloads, and ~2413x throughput per watt and ~94.5x latency improvements over a multi-core Intel E5-2620 CPU. Finally, we show that the NCAM is not limited to just kNN and can be generalized to act as a content addressable memory (CAM) or ternary-CAM (TCAM).

Motivation & Objective

• Address the performance bottleneck in kNN search caused by excessive data movement in modern applications like NLP, vision, and robotics.
• Overcome the limitations of approximate kNN methods, which degrade into linear search in high-dimensional spaces and sacrifice accuracy.
• Design a scalable, energy-efficient solution for exact nearest neighbor search using 3D die-stacking and processing-in-memory (PIM) technology.
• Demonstrate that NCAM can outperform state-of-the-art CPUs and GPUs in both energy efficiency and latency for kNN workloads.
• Generalize NCAM beyond kNN to function as a content addressable memory (CAM) or ternary-CAM (TCAM) for broader in-memory computing applications.

Proposed method

• Design a 3D-stacked NCAM architecture that integrates processing logic with DRAM using die-stacking to minimize data movement.
• Implement in-memory computation to perform distance calculations directly within memory, avoiding off-chip data transfers.
• Leverage emerging memory technologies and interfaces to achieve higher density and bandwidth, enabling efficient kNN computation.
• Use post-placement and route estimation to model the physical design and performance of NCAM, ensuring compatibility with real-world substrates.
• Integrate NCAM with existing memory subsystems via modular design, avoiding direct integration with DRAM dies.
• Extend NCAM’s functionality to support general-purpose CAM and TCAM operations, enabling broader use in content-based data retrieval.

Experimental results

Research questions

• RQ1 Can a near-data processing architecture achieve exact kNN search with significantly lower latency and higher energy efficiency than modern CPUs and GPUs?
• RQ2 How does NCAM’s performance scale under typical mobile and server workloads compared to state-of-the-art software kNN implementations?
• RQ3 To what extent do approximate kNN methods degrade in high-dimensional spaces, and can NCAM eliminate this trade-off?
• RQ4 Is the NCAM design compatible with emerging memory substrates in terms of power and physical integration?
• RQ5 Can NCAM be generalized beyond kNN to serve as a general-purpose content addressable memory (CAM) or ternary-CAM (TCAM)?

Key findings

• NCAM achieves up to 160x higher throughput per watt than a NVIDIA Titan X GPU for server workloads, demonstrating superior energy efficiency.
• NCAM reduces latency by three orders of magnitude (1000x) compared to a NVIDIA Titan X GPU in server workloads.
• For multi-core CPUs, NCAM delivers approximately 2413x higher throughput per watt and 94.5x lower latency than an Intel E5-2620 CPU.
• The performance gains are attributed to eliminating data movement through in-memory processing, enabling exact kNN search without accuracy loss.
• NCAM’s design is compatible with emerging memory substrates and supports modular integration via 3D die-stacking technology.
• NCAM can be generalized to function as a CAM or TCAM, extending its applicability beyond kNN to content-based memory access.