[Paper Review] A Hierarchical Distributed Processing Framework for Big Image Data

Le Dong, Zhiyu Lin+6 more|arXiv (Cornell University)|Jul 3, 2016

Brain Tumor Detection and Classification3 citations

TL;DR

This paper proposes ICP, a hierarchical distributed processing framework for big image data that leverages MapReduce with novel data representations—P-Image and Big-Image—to enable efficient parallel processing. SICP handles pre-stored static data, while DICP supports real-time dynamic input, achieving significant speedups over single-node methods like OpenCV without sacrificing result quality, as validated on ImageNet with high scalability and pressure resistance.

ABSTRACT

This paper introduces an effective processing framework nominated ICP (Image Cloud Processing) to powerfully cope with the data explosion in image processing field. While most previous researches focus on optimizing the image processing algorithms to gain higher efficiency, our work dedicates to providing a general framework for those image processing algorithms, which can be implemented in parallel so as to achieve a boost in time efficiency without compromising the results performance along with the increasing image scale. The proposed ICP framework consists of two mechanisms, i.e. SICP (Static ICP) and DICP (Dynamic ICP). Specifically, SICP is aimed at processing the big image data pre-stored in the distributed system, while DICP is proposed for dynamic input. To accomplish SICP, two novel data representations named P-Image and Big-Image are designed to cooperate with MapReduce to achieve more optimized configuration and higher efficiency. DICP is implemented through a parallel processing procedure working with the traditional processing mechanism of the distributed system. Representative results of comprehensive experiments on the challenging ImageNet dataset are selected to validate the capacity of our proposed ICP framework over the traditional state-of-the-art methods, both in time efficiency and quality of results.

Motivation & Objective

• To address the growing challenge of processing massive-scale image data efficiently in distributed and cloud environments.
• To overcome the limitations of single-node image processing frameworks like OpenCV, which become bottlenecks as data volume increases.
• To design a general-purpose, scalable framework that supports both static and dynamic image workloads in cloud computing environments.
• To improve time efficiency without compromising result quality by leveraging MapReduce and novel data abstractions.
• To enable robust, high-performance processing of big image data through optimized data representation and distributed scheduling.

Proposed method

• Proposes ICP, a hierarchical framework with two mechanisms: SICP for static big image data and DICP for dynamic input.
• Introduces P-Image and Big-Image as novel data representations to minimize I/O overhead and improve data locality in distributed systems.
• Uses MapReduce as the underlying execution model to enable horizontal scaling and fault tolerance across clusters.
• Employs parallel processing pipelines in DICP to allow immediate response to incoming image streams without delay.
• Designs an internal scheduling mechanism in DICP to ensure stability and pressure resistance under high load.
• Enables concurrent processing of image generation, P-Image/Big-Image creation, and actual image processing to avoid pipeline bottlenecks.

Experimental results

Research questions

• RQ1 How can a general-purpose distributed framework be designed to efficiently process large-scale image data without sacrificing result quality?
• RQ2 What novel data representations can significantly reduce I/O overhead and improve processing efficiency in distributed image processing?
• RQ3 How can a framework support both pre-stored static image data and real-time dynamic image input in a unified, scalable architecture?
• RQ4 To what extent can the proposed framework outperform single-node systems like OpenCV in terms of time efficiency and scalability?
• RQ5 How does the internal scheduling mechanism in DICP affect performance under high load and dynamic workloads?

Key findings

• The ICP framework achieves substantial speedups over single-node systems like OpenCV, especially for large-scale image processing workloads.
• P-Image and Big-Image representations significantly reduce input consumption and I/O overhead, improving overall processing efficiency.
• SICP demonstrates high efficiency in processing large-scale static image data stored in distributed systems, with performance scaling linearly with cluster size.
• DICP exhibits strong pressure resistance and stability under high workloads, maintaining consistent performance even with increasing image input rates.
• The framework maintains high result quality, as validated by visual and quantitative comparisons with state-of-the-art methods on the ImageNet dataset.
• The experimental results confirm that the proposed framework is both scalable and robust, suitable for real-world big image data processing in cloud environments.