[Paper Review] User Mode Memory Page Allocation: A Silver Bullet For Memory Allocation?

Niall Douglas|arXiv (Cornell University)|May 9, 2011

Parallel Computing and Optimization Techniques25 references4 citations

TL;DR

This paper proposes a user-mode page allocator that shifts memory page allocation from the kernel to user space, enabling direct, bare-metal access to MMU page mappings. Benchmarks show up to 10x faster allocation for 0–8MB blocks and 4.5x faster resizing, with real-world applications improving by up to 5.68%, suggesting a potential 100x performance gain with full OS support and a new allocator API.

ABSTRACT

This paper proposes a novel solution: the elimination of paged virtual memory and partial outsourcing of memory page allocation and manipulation from the operating system kernel into the individual process' user space - a user mode page allocator - which allows an application to have direct, bare metal access to the page mappings used by the hardware Memory Management Unit (MMU) for its part of the overall address space. A user mode page allocator based emulation of the mmap() abstraction layer of dlmalloc is then benchmarked against the traditional kernel mode implemented mmap() in a series of synthetic Monte-Carlo and real world application settings. Given the superb synthetic and positive real world results from the profiling conducted, this paper proposes that with proper operating system and API support one could gain a further order higher performance again while keeping allocator performance invariant to the amount of memory being allocated or freed i.e. a 100x performance improvement or more in some common use cases. It is rare that through a simple and easy to implement API and operating system structure change one can gain a Silver Bullet with the potential for a second one.

Motivation & Objective

• Address the growing performance bottleneck caused by kernel-mode memory allocation as memory capacity outpaces memory speed.
• Investigate whether moving page allocation to user space can eliminate kernel traps and reduce allocation overhead.
• Evaluate the feasibility and performance of a user-mode page allocator as a replacement for traditional kernel-managed mmap() calls.
• Explore the potential for a new, scale-invariant allocator API that supports batch allocation and avoids address reallocation.
• Determine whether user-mode page allocation can enable a new generation of high-performance memory allocators for modern, memory-intensive workloads.

Proposed method

• Design and implement a user-mode page allocator that emulates the mmap() system call interface using user-space memory management.
• Benchmark the user-mode allocator against the standard kernel-mode mmap() in synthetic Monte-Carlo workloads and real-world applications.
• Use binary patching to replace the system allocator in unmodified binaries with the user-mode allocator to simulate real-world deployment.
• Measure performance across varying block sizes (0–8MB), allocation patterns, and memory access characteristics.
• Analyze cache behavior and page fault effects to assess whether reduced cache pollution contributes to performance gains.
• Propose and reference the N1527 C1X malloc API extension, which supports batch allocation and hints to improve allocator efficiency.

Experimental results

Research questions

• RQ1 Can user-mode page allocation eliminate the performance penalty of kernel traps during memory allocation?
• RQ2 Does user-mode page allocation achieve scale-invariant performance across different memory block sizes?
• RQ3 What is the impact of reduced cache pollution from avoiding page fault handlers on allocation performance?
• RQ4 To what extent can user-mode page allocation improve real-world application performance without source code changes?
• RQ5 How does the proposed user-mode approach compare to traditional kernel-managed allocation in terms of fragmentation, locality, and scalability?

Key findings

• The user-mode page allocator achieved scale-invariant performance up to 1MB block sizes, indicating consistent behavior regardless of allocation size.
• For block sizes between 0–8MB, the user-mode allocator delivered up to a 10x performance improvement over kernel-mode mmap() calls.
• Block resizing performance improved by up to 4.5x with the user-mode allocator, suggesting reduced overhead in reallocation operations.
• A 2x performance gain was observed for very small allocations under non-paged memory, possibly due to reduced cache pollution from avoiding page fault handlers.
• In real-world testing with binary-patched applications, all but one scenario showed performance improvement, with a mean gain of +1.88% and a median of +1.20%.
• The study suggests that with full OS and API support, a further 10x to 100x performance improvement could be achieved, especially in workloads involving frequent or large allocations.