[Paper Review] Optimal Sequential Decision-Making in Geosteering: A Reinforcement Learning Approach

Ressi Bonti Muhammad, Sergey Alyaev+1 more|arXiv (Cornell University)|Oct 7, 2023

Oil and Gas Production Techniques4 citations

TL;DR

This paper proposes a model-free reinforcement learning (RL) approach using Deep Q-Networks (DQN) to optimize sequential geosteering decisions in real time, bypassing the need for explicit environment models. The method achieves performance comparable to Approximate Dynamic Programming (ADP) with drastically reduced inference time—evaluating 1,000 reservoir realizations in under 10 seconds versus 4–5 hours for ADP—demonstrating computational efficiency and adaptability to complex, uncertain geosteering environments.

ABSTRACT

Trajectory adjustment decisions throughout the drilling process, called geosteering, affect subsequent choices and information gathering, thus resulting in a coupled sequential decision problem. Previous works on applying decision optimization methods in geosteering rely on greedy optimization or approximate dynamic programming (ADP). Either decision optimization method requires explicit uncertainty and objective function models, making developing decision optimization methods for complex and realistic geosteering environments challenging to impossible. We use the Deep Q-Network (DQN) method, a model-free reinforcement learning (RL) method that learns directly from the decision environment, to optimize geosteering decisions. The expensive computations for RL are handled during the offline training stage. Evaluating DQN needed for real-time decision support takes milliseconds and is faster than the traditional alternatives. Moreover, for two previously published synthetic geosteering scenarios, our results show that RL achieves high-quality outcomes comparable to the quasi-optimal ADP. Yet, the model-free nature of RL means that by replacing the training environment, we can extend it to problems where the solution to ADP is prohibitively expensive to compute. This flexibility will allow applying it to more complex environments and make hybrid versions trained with real data in the future.

Motivation & Objective

• Address the challenge of sequential decision-making in geosteering, where trajectory adjustments depend on evolving LWD data and uncertain subsurface conditions.
• Overcome limitations of existing methods like greedy optimization and Approximate Dynamic Programming (ADP), which require explicit environment models and become infeasible in complex scenarios.
• Develop a flexible, computationally efficient decision support system for geosteering using model-free reinforcement learning that learns directly from environment interactions.
• Demonstrate that RL can achieve quasi-optimal outcomes comparable to ADP while reducing evaluation time from hours to seconds.
• Enable extension to more complex or previously intractable geosteering problems by replacing the training environment without retraining the entire policy.

Proposed method

• Employ Deep Q-Networks (DQN), a model-free reinforcement learning algorithm, to learn optimal geosteering policies directly from simulated drilling environments.
• Define the state space using sensor inputs and geological parameters (e.g., reservoir boundaries, wellbore position, inclination), excluding explicit posterior updates in one variant (RL-Sensor) to reduce computation.
• Use a reward function based on expected hydrocarbon production and proximity to reservoir targets, encouraging trajectory decisions that maximize long-term value.
• Train the DQN agent offline using experience replay and target networks to stabilize learning, with rewards shaped to reflect geosteering objectives.
• Decouple the Bayesian model update step by using raw sensor inputs as state features, reducing computational overhead compared to full posterior modeling.
• Evaluate the trained agent in real-time inference mode, achieving inference times in milliseconds, suitable for operational deployment.

Experimental results

Research questions

• RQ1 Can a model-free reinforcement learning approach achieve performance comparable to quasi-optimal ADP in synthetic geosteering scenarios without requiring explicit environment models?
• RQ2 How does the computational efficiency of RL compare to traditional ADP-based methods when evaluating multiple reservoir realizations?
• RQ3 What is the impact of omitting Bayesian posterior updates in the state representation (i.e., RL-Sensor vs. RL-Posterior) on performance and inference speed?
• RQ4 To what extent can RL generalize across different geosteering environments with minimal architectural or hyperparameter changes?
• RQ5 How does RL handle sequential decision-making under uncertainty when sensor data is noisy or incomplete, and what are the implications for real-world deployment?

Key findings

• The RL-Posterior method achieved a 19–33% improvement in value function compared to greedy optimization across different synthetic scenarios.
• The RL-Sensor method reduced computation time from 2,500 seconds to 500 seconds while yielding slightly better rewards than the RL-Posterior variant.
• For the second scenario, RL achieved rewards comparable to the quasi-optimal DSDP (Approximate Dynamic Programming) solution, with no statistically significant difference.
• RL inference evaluated 1,000 reservoir realizations in under 10 seconds, whereas DSDP required 4–5 hours for the same task, demonstrating a 200x speedup in evaluation.
• The DQN agent required only 20 minutes of offline training to achieve real-time inference performance, making it suitable for operational deployment.
• The model-free nature of RL allows seamless adaptation to new geosteering environments by retraining on new simulation data without redesigning the underlying decision framework.