[Paper Review] Numerical Simulation of Multi-phase Flow in Porous Media on Parallel Computers

Hui Liu, Shen, Lihua+3 more|arXiv (Cornell University)|Jun 2, 2016

Advanced Numerical Methods in Computational Mathematics20 references3 citations

TL;DR

This paper presents a parallel black oil reservoir simulator designed for large-scale multi-phase flow simulations in porous media, leveraging MPI-based domain decomposition and advanced linear solvers. It achieves excellent scalability on exascale-class systems, demonstrating linear and even super-linear performance on models with up to 140 million grid cells.

ABSTRACT

A parallel reservoir simulator has been developed, which is designed for large-scale black oil simulations. It handles three phases, including water, oil and gas, and three components, including water, oil and gas. This simulator can calculate traditional reservoir models and naturally fractured models. Various well operations are supported, such as water flooding, gas flooding and polymer flooding. The operation constraints can be fixed bottom-hole pressure, a fixed fluid rate, and combinations of them. The simulator is based on our in-house platform, which provides grids, cell-centred data, linear solvers, preconditioners and well modeling. The simulator and the platform use MPI for communications among computation nodes. Our simulator is capable of simulating giant reservoir models with hundreds of millions of grid cells. Numerical simulations show that our simulator matches with commercial simulators and it has excellent scalability.

Motivation & Objective

• To develop a scalable parallel reservoir simulator capable of handling large-scale black oil and oil-water flow simulations in complex, heterogeneous reservoirs.
• To support multiple phase models (water, oil, gas) and component-based simulations with full well operation constraints.
• To ensure robustness and efficiency in solving large-scale linear systems arising from implicit discretization of multiphase flow equations.
• To demonstrate linear and super-linear scalability on high-performance computing platforms for billion-cell reservoir models.
• To validate the simulator against commercial tools and establish its reliability for industrial-scale applications.

Proposed method

• The simulator is built on an in-house parallel platform using MPI for inter-node communication, enabling distributed memory computation across thousands of processors.
• It employs a cell-centered finite difference method with implicit time stepping and inexact Newton methods for solving the nonlinear system of equations.
• The pressure equation is solved using BiCGSTAB iterative linear solvers with RAS (Restricted Additive Schwarz) preconditioning for enhanced convergence.
• The model supports three-phase flow (water, oil, gas) with three components, including solution gas and free gas, using Darcy’s law and mass conservation equations.
• The simulator handles both regular and naturally fractured reservoirs using dual-porosity and dual-permeability formulations.
• Well models are implemented with fixed bottom-hole pressure, fixed rate, and mixed constraints, enabling simulation of water, gas, and polymer flooding.

Experimental results

Research questions

• RQ1 Can a parallel reservoir simulator achieve linear scalability on large-scale, multi-phase reservoir models with hundreds of millions of grid cells?
• RQ2 How does the performance of the inexact Newton method and BiCGSTAB linear solver scale with increasing numbers of MPI processes?
• RQ3 To what extent does the RAS preconditioner improve convergence and scalability in solving large sparse linear systems from reservoir simulation?
• RQ4 How does the simulator compare in accuracy and performance to commercial simulators on benchmark reservoir models?
• RQ5 Can the simulator handle complex reservoir features such as fractures, faults, and polymer flooding with high computational efficiency?

Key findings

• The simulator achieved linear scalability on the IBM Blue Gene/Q system for a 140-million-cell model, reducing simulation time from ~11.4 hours (256 cores) to ~0.4 hours (4096 cores).
• For a 100-million-cell model, the simulator showed super-linear scalability between 1024 and 2048 MPI tasks, with a 2.3x speedup from 2048 to 4096 cores.
• The inexact Newton method remained robust across all cases, with average nonlinear iterations between 101 and 145, and convergence maintained even at high processor counts.
• The linear solver (BiCGSTAB) showed increasing average iterations from 4.9 to 6.2 as the number of MPI tasks increased, yet overall runtime decreased due to parallelism.
• A 3-billion-degree-of-freedom linear system was solved in under 2 minutes on 4096 cores using GMRES(30) with RAS preconditioning, demonstrating extreme scalability.
• Numerical results matched those from commercial simulators, validating the accuracy and reliability of the proposed parallel implementation.