[Paper Review] Big Data Spark Solution for Functional Magnetic Resonance Imaging

Saman Sarraf, Mehdi Ostadhashem|arXiv (Cornell University)|Mar 23, 2016

Functional Brain Connectivity Studies13 references4 citations

TL;DR

This paper proposes a PySpark-based pipeline for accelerating functional MRI (fMRI) data processing by leveraging in-memory, distributed computing to extract brain networks—specifically the Default Mode Network (DMN)—using sum of squared differences (SSD) template matching. The solution achieves a 3.7x speedup over pure Python on a single node, with potential for 10–20x improvement on clusters, while maintaining accuracy and enabling flexible result storage in multiple formats.

ABSTRACT

Recently, Big Data applications have rapidly expanded into different industries. Healthcare is also one the industries willing to use big data platforms so that some big data analytics tools have been adopted in this field to some extent. Medical imaging which is a pillar in diagnostic healthcare deals with high volume of data collection and processing. A huge amount of 3D and 4D images are acquired in different forms and resolutions using a variety of medical imaging modalities. Preprocessing and analyzing imaging data is currently a long process and cost and time consuming. However, not many big data platforms have been provided or redesigned for medical imaging purposes because of some restrictions such as data format. In this paper, we designed, developed and successfully tested a new pipeline for medical imaging data (especially functional magnetic resonance imaging - fMRI) using Big Data Spark / PySpark platform on a single node which allows us to read and load imaging data, convert them to Resilient Distributed Datasets in order manipulate and perform in-memory data processing in parallel and convert final results to imaging format while the pipeline provides an option to store the results in other formats such as data frame. Using this new solution and pipeline, we repeated our previous works in which we extracted brain networks from fMRI data using template matching and sum of squared differences (SSD) method. The final results revealed our Spark (PySpark) based solution improved the performance (in terms of processing time) around 4 times on a single compared to the previous work developed in Python.

Motivation & Objective

• To address the performance bottleneck in fMRI data preprocessing and analysis, which is time-consuming and computationally expensive.
• To integrate big data technologies like Apache Spark with medical imaging workflows to enable scalable, parallel processing of large neuroimaging datasets.
• To develop a portable, extensible pipeline that supports both in-memory computation and multi-format result storage (e.g., data frames, NIfTI).
• To evaluate the performance gain of using PySpark over traditional Python implementations for template-based brain network extraction.
• To demonstrate the feasibility of adapting big data platforms to neuroimaging workloads despite data format and structural constraints.

Proposed method

• The pipeline loads 84 fMRI components in NIfTI format into memory using the nibabel library.
• Imaging data and a pre-defined DMN template are converted into Resilient Distributed Datasets (RDDs), the core data abstraction in Apache Spark.
• SSD-based template matching is implemented using PySpark’s flatMap and zip operations on RDDs to compute similarity across spatial coordinates.
• The SSD equation is applied in parallel: $SSD = \sum_{x,y} [f(x,y) - t(x-u,y-v)]^2$, enabling distributed computation across partitions.
• Results are converted back to standard imaging formats (e.g., NIfTI) or stored as data frames for interoperability with other big data tools.
• Performance is benchmarked on a standalone single-node Spark setup, comparing PySpark execution time to native Python implementation.

Experimental results

Research questions

• RQ1 Can a PySpark-based pipeline significantly reduce processing time for fMRI data analysis compared to traditional Python workflows?
• RQ2 How does in-memory, distributed processing via RDDs impact the performance of template-based brain network extraction in fMRI data?
• RQ3 To what extent can a single-node Spark implementation accelerate SSD-based template matching in neuroimaging?
• RQ4 Can the pipeline maintain accuracy while enabling flexible output formats beyond standard neuroimaging formats?
• RQ5 What is the potential for further performance gains when scaling the pipeline to multi-node clusters or optimizing for parallel execution?

Key findings

• The PySpark-based pipeline reduced processing time from 23.87 seconds in pure Python to 6.44 seconds, achieving a 3.7x speedup on a single node.
• The performance gain is attributed to in-memory computation and parallel processing via Spark’s RDD abstraction, even in a single-node setup.
• The pipeline maintains the same accuracy as the original Python implementation, confirming that performance improvements do not compromise result fidelity.
• The solution supports flexible result storage, allowing outputs to be saved as data frames or standard imaging formats like NIfTI.
• The authors project a 10–20x performance improvement when scaling to high-performance clusters and optimizing for full parallelism.
• The study demonstrates that big data platforms like Spark can be effectively adapted to neuroimaging workloads despite data format and domain-specific constraints.