[Paper Review] A Cloud-based Service for Real-Time Performance Evaluation of NoSQL Databases

Omar Almootassem, Syed Hamza Husain+2 more|arXiv (Cornell University)|May 23, 2017

Cloud Computing and Resource Management8 citations

TL;DR

This paper presents a cloud-based web service that enables real-time, interactive performance evaluation of NoSQL databases such as MongoDB, DynamoDB, CouchDB, and Firebase. By allowing users to select test types and target databases, the system executes benchmarks in real time and displays results alongside historical data and geographic test locations, offering a dynamic alternative to static benchmarking reports.

ABSTRACT

We have created a cloud-based service that allows the end users to run tests on multiple different databases to find which databases are most suitable for their project. From our research, we could not find another application that enables the user to test several databases to gauge the difference between them. This application allows the user to choose which type of test to perform and which databases to target. The application also displays the results of different tests that were run by other users previously. There is also a map to show the location where all the tests are run to give the user an estimate of the location. Unlike the orthodox static tests and reports conducted to evaluate NoSQL databases, we have created a web application to run and analyze these tests in real time. This web application evaluates the performance of several NoSQL databases. The databases covered are MongoDB, DynamoDB, CouchDB, and Firebase. The web service is accessible from: nosqldb.nextproject.ca.

Motivation & Objective

• To address the lack of interactive, real-time tools for comparing NoSQL database performance across multiple systems.
• To provide a dynamic platform where users can run custom performance tests on multiple NoSQL databases simultaneously.
• To aggregate and display results from previous tests to support data-driven database selection.
• To enhance transparency by showing the geographic distribution of test executions for latency estimation.
• To offer a live, web-based interface that outperforms traditional static benchmarking reports in usability and responsiveness.

Proposed method

• The system is implemented as a cloud-hosted web application accessible via a public URL.
• Users can select from predefined test types (e.g., insert, read, update, delete) and choose target NoSQL databases.
• Tests are executed in real time using cloud-based infrastructure, ensuring consistent and reproducible conditions.
• Performance metrics (e.g., latency, throughput) are collected and visualized immediately after each test run.
• Historical test results are stored and made publicly available for comparison across users and configurations.
• A geographical map displays the physical locations of test execution nodes to help estimate network latency.

Experimental results

Research questions

• RQ1 How can real-time performance evaluation of NoSQL databases be made interactive and accessible to end users?
• RQ2 What are the performance differences between popular NoSQL databases under common workloads?
• RQ3 Can a shared, cloud-based platform improve the transparency and reproducibility of NoSQL benchmarking?
• RQ4 How does the geographic distribution of test nodes affect perceived database performance?
• RQ5 To what extent can live, user-driven testing outperform static, precomputed benchmark reports?

Key findings

• The service enables real-time performance evaluation of four major NoSQL databases: MongoDB, DynamoDB, CouchDB, and Firebase.
• Users can run and compare tests across multiple databases with configurable workloads and receive immediate feedback.
• Historical test results are publicly accessible, allowing users to compare performance trends and make informed decisions.
• The system provides a geographic map showing the physical locations of test execution nodes, aiding in latency estimation.
• The platform offers a dynamic alternative to static benchmarking, improving usability and real-world relevance.
• No specific quantitative performance metrics (e.g., latency values or throughput rates) are reported in the abstract.