[Paper Review] RL-Controller: a reinforcement learning framework for active structural control

Soheila Sadeghi Eshkevari, Soheil Sadeghi Eshkevari+2 more|arXiv (Cornell University)|Mar 13, 2021

Elevator Systems and Control34 references4 citations

TL;DR

This paper introduces RL-Controller, a model-free reinforcement learning framework for active structural control that trains agents to minimize interstory drifts in buildings under seismic loads. Using a data-driven approach with deep reinforcement learning, it achieves 25% lower average interstory drifts and 26% lower accelerations than LQG control, while using less control force energy, demonstrating superior performance without prior system knowledge.

ABSTRACT

To maintain structural integrity and functionality during the designed life cycle of a structure, engineers are expected to accommodate for natural hazards as well as operational load levels. Active control systems are an efficient solution for structural response control when a structure is subjected to unexpected extreme loads. However, development of these systems through traditional means is limited by their model dependent nature. Recent advancements in adaptive learning methods, in particular, reinforcement learning (RL), for real-time decision making problems, along with rapid growth in high-performance computational resources, help structural engineers to transform the classic model-based active control problem to a purely data-driven one. In this paper, we present a novel RL-based approach for designing active controllers by introducing RL-Controller, a flexible and scalable simulation environment. The RL-Controller includes attributes and functionalities that are defined to model active structural control mechanisms in detail. We show that the proposed framework is easily trainable for a five story benchmark building with 65% reductions on average in inter story drifts (ISD) when subjected to strong ground motions. In a comparative study with LQG active control method, we demonstrate that the proposed model-free algorithm learns more optimal actuator forcing strategies that yield higher performance, e.g., 25% more ISD reductions on average with respect to LQG, without using prior information about the mechanical properties of the system.

Motivation & Objective

• To develop a flexible, scalable, and model-free active control framework for structural systems that overcomes limitations of traditional model-dependent control methods.
• To enable real-time, adaptive control of buildings under seismic excitations using reinforcement learning without requiring prior knowledge of structural mechanical properties.
• To improve structural resilience by minimizing interstory drifts, accelerations, and base shear during strong ground motions.
• To demonstrate the superiority of data-driven RL control over classical optimal control methods like LQG in complex, uncertain, and dynamic environments.
• To validate the framework on a five-story benchmark building using diverse historical earthquake records.

Proposed method

• The framework models active structural control as a Markov Decision Process (MDP), with the agent (neural network) selecting control forces based on real-time structural state observations.
• A deep Q-network (DQN) with experience replay and target networks is used to stabilize training and approximate the optimal action-value function in continuous state-action spaces.
• The environment simulates a five-story benchmark building with base excitation from historical earthquakes, including both white noise and impulse-like ground motions for robust training.
• The reward function is designed to penalize high interstory drifts and accelerations while minimizing control force energy, promoting balanced performance.
• The RL agent is trained using a combination of 7 diverse earthquake records (e.g., Northridge, Kobe, Loma Prieta) to ensure generalization across varying ground motion characteristics.
• The framework is compared against the LQG control method under identical conditions to evaluate performance across multiple metrics.

Experimental results

Research questions

• RQ1 Can a model-free reinforcement learning framework outperform classical model-based control methods like LQG in minimizing structural response under diverse seismic excitations?
• RQ2 How does the inclusion of impulse-like and white-noise ground motions during training affect the robustness and performance of the RL-based controller?
• RQ3 To what extent can the RL-Controller reduce interstory drifts and accelerations without prior knowledge of the structure’s mechanical properties?
• RQ4 How does the control force energy and base shear compare between the RL-Controller and LQG under real-world earthquake scenarios?
• RQ5 Can the RL-Controller generalize across a wide range of ground motion parameters (amplitude, frequency, duration) without retraining?

Key findings

• The RL-Controller achieved an average 25% reduction in interstory drift (ISD) compared to LQG control across seven historical earthquakes, demonstrating superior vibration mitigation.
• The RL-Controller reduced average story accelerations by 26% compared to LQG, indicating better control of inertial forces and improved occupant comfort.
• The RL-Controller applied 30% less control force energy on average than LQG, as measured by normalized energy of the control signal, indicating greater energy efficiency.
• The maximum story shear was significantly reduced under RL-Controller, with a 20% lower ratio of maximum controlled to uncontrolled shear compared to LQG, indicating improved structural safety.
• The RL-Controller outperformed LQG across all seven earthquake records, with consistent improvements in all four performance metrics (J1–J4), showing strong generalization and robustness.
• The framework successfully learned to balance displacement control and acceleration mitigation, avoiding the trade-off typically seen in control systems, by absorbing shock impulses effectively.