[Paper Review] COSIME: FeFET based Associative Memory for In-Memory Cosine Similarity Search

Che-Kai Liu, Haobang Chen+9 more|arXiv (Cornell University)|Jul 25, 2022

Ferroelectric and Negative Capacitance Devices4 citations

TL;DR

COSIME proposes a ferroelectric FET (FeFET)-based in-memory associative memory for efficient cosine similarity search (CSS), leveraging current-mode analog circuits and winner-take-all (WTA) logic to perform parallel, low-energy CSS across stored vectors. It achieves 333× latency reduction and 90.5× energy improvement over an approximated CSS design, with 47.1× speedup and 98.5× energy efficiency gain over GPU in HDC inference.

ABSTRACT

In a number of machine learning models, an input query is searched across the trained class vectors to find the closest feature class vector in cosine similarity metric. However, performing the cosine similarities between the vectors in Von-Neumann machines involves a large number of multiplications, Euclidean normalizations and division operations, thus incurring heavy hardware energy and latency overheads. Moreover, due to the memory wall problem that presents in the conventional architecture, frequent cosine similarity-based searches (CSSs) over the class vectors requires a lot of data movements, limiting the throughput and efficiency of the system. To overcome the aforementioned challenges, this paper introduces COSIME, an general in-memory associative memory (AM) engine based on the ferroelectric FET (FeFET) device for efficient CSS. By leveraging the one-transistor AND gate function of FeFET devices, current-based translinear analog circuit and winner-take-all (WTA) circuitry, COSIME can realize parallel in-memory CSS across all the entries in a memory block, and output the closest word to the input query in cosine similarity metric. Evaluation results at the array level suggest that the proposed COSIME design achieves 333X and 90.5X latency and energy improvements, respectively, and realizes better classification accuracy when compared with an AM design implementing approximated CSS. The proposed in-memory computing fabric is evaluated for an HDC problem, showcasing that COSIME can achieve on average 47.1X and 98.5X speedup and energy efficiency improvements compared with an GPU implementation.

Motivation & Objective

• Address the high energy and latency overhead of cosine similarity search (CSS) in von Neumann architectures due to numerous multiplications, normalizations, and divisions.
• Overcome the memory wall problem in traditional systems by reducing data movement through in-memory computation.
• Design a general-purpose, energy-efficient, and accurate CSS accelerator for machine learning workloads, particularly for hyperdimensional computing (HDC) and DNN inference.
• Achieve high accuracy comparable to full-precision software CSS while maintaining hardware efficiency.
• Enable scalable and robust in-memory CSS using non-volatile memory (NVM) technologies beyond FeFETs, ensuring broad applicability.

Proposed method

• Utilizes two FeFET arrays: one for computing row-wise dot products between input query and stored class vectors, and another for bit counting to estimate L2 norms.
• Employs current-mode translinear analog circuits to compute squared L2 norms and perform division operations required for cosine similarity.
• Integrates a winner-take-all (WTA) circuit to identify the vector with maximum cosine similarity to the input query in parallel.
• Leverages the one-transistor AND gate functionality of FeFETs to enable non-volatile storage and in-memory computation.
• Designs peripheral circuits independent of the NVM array, enabling compatibility with other NVMs (e.g., ReRAM) as long as output currents are within sensing range.
• Employs analog computation to avoid digital arithmetic bottlenecks, reducing area and energy consumption.

Experimental results

Research questions

• RQ1 Can an in-memory associative memory based on FeFETs achieve high-accuracy cosine similarity search with low latency and energy consumption?
• RQ2 How does the proposed COSIME design compare in performance and accuracy to existing approximate CSS accelerators using Hamming distance or approximated cosine metrics?
• RQ3 To what extent does COSIME reduce data movement and improve throughput in memory-bound workloads like HDC inference?
• RQ4 How robust is COSIME to device variations and analog circuit errors in real-world deployment?
• RQ5 Can the COSIME architecture be generalized to other NVM technologies beyond FeFETs?

Key findings

• COSIME achieves 333× speedup and 90.5× energy reduction compared to an existing approximated CSS design using the same FeFET technology.
• In HDC classification tasks, COSIME delivers 47.1× higher throughput and 98.5× better energy efficiency than an NVIDIA 1080 GPU at 1,000-dimensional hypervectors.
• COSIME maintains high classification accuracy—on average 7% better than Hamming distance-based methods—by enabling exact cosine similarity computation.
• The system is robust to device variation, with HDC classification tolerating up to 10% error rate in the AM, which is within HDC’s inherent error tolerance.
• The design is generalizable to other NVMs with access transistors, as peripheral circuits are independent of the memory array technology.
• Higher-dimensional and larger-class datasets (e.g., ISOLET with 26 classes) benefit most from COSIME, showing the greatest speedup and energy efficiency gains.