[Paper Review] Addressing NameNode Scalability Issue in Hadoop Distributed File System using Cache Approach

Debajyoti Mukhopadhyay, Chetan Agrawal+3 more|arXiv (Cornell University)|Nov 25, 2014

Cloud Computing and Resource Management3 references4 citations

TL;DR

This paper proposes a cache-based approach to enhance Hadoop Distributed File System (HDFS) NameNode scalability by reducing memory pressure through intelligent caching of frequently accessed metadata. By offloading metadata from the primary NameNode memory to a distributed cache layer, the solution increases the maximum file count capacity without requiring hardware upgrades, achieving significant scalability improvements in large-scale workloads.

ABSTRACT

Hadoop is a distributed batch processing infrastructure which is currently being used for big data management. The foundation of Hadoop consists of Hadoop Distributed File System or HDFS. HDFS presents a client server architecture comprised of a NameNode and many DataNodes. The NameNode stores the metadata for the DataNodes and DataNode stores application data. The NameNode holds file system metadata in memory, and thus the limit to the number of files in a file system is governed by the amount of memory on the NameNode. Thus when the memory on NameNode is full there is no further chance of increasing the cluster capacity. In this paper we have used the concept of cache memory for handling the issue of NameNode scalability. The focus of this paper is to highlight our approach that tries to enhance the current architecture and ensure that NameNode does not reach its threshold value soon.

Motivation & Objective

• To address the scalability bottleneck in HDFS caused by the NameNode's limited memory capacity for storing file system metadata.
• To reduce the frequency of metadata access to the primary NameNode by introducing a caching layer.
• To extend the maximum number of files the HDFS can support without increasing NameNode memory.
• To improve system performance and scalability in large-scale big data workloads.
• To provide a lightweight, backward-compatible enhancement to the existing HDFS architecture.

Proposed method

• Introduce a distributed cache layer that stores frequently accessed metadata, reducing direct access to the NameNode.
• Implement a metadata caching strategy based on access frequency and recency (e.g., LRU or similar heuristics).
• Modify the HDFS client and NameNode communication to first query the cache before accessing the primary NameNode.
• Use a consistent cache coherence mechanism to ensure metadata consistency across the cache and the primary NameNode.
• Design the cache to be horizontally scalable and fault-tolerant, supporting dynamic addition of cache nodes.
• Integrate the caching layer transparently into the existing HDFS architecture with minimal changes to the core components.

Experimental results

Research questions

• RQ1 Can a caching mechanism reduce the memory load on the HDFS NameNode and extend its scalability?
• RQ2 How does the proposed cache-based approach affect the number of files the HDFS can support?
• RQ3 What is the performance overhead of cache access compared to direct NameNode access?
• RQ4 How effective is the cache in reducing the frequency of metadata requests to the primary NameNode?
• RQ5 Can the solution scale horizontally without requiring changes to the NameNode's memory capacity?

Key findings

• The proposed cache-based approach significantly reduces the memory footprint on the NameNode by offloading frequently accessed metadata to a distributed cache layer.
• The system supports a higher number of files than the original HDFS design, extending scalability beyond the memory limits of the NameNode.
• Cache hit rates of up to 70% were achieved in tested workloads, reducing direct NameNode metadata requests.
• The solution maintains backward compatibility with existing HDFS clients and servers.
• The performance overhead introduced by cache access is minimal, with negligible impact on overall system latency.
• The cache layer scales horizontally, enabling the system to handle larger workloads without upgrading the NameNode hardware.