[Paper Review] DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances

Tianming Wang, Wenjie Lu+2 more|arXiv (Cornell University)|Jul 10, 2019

Reinforcement Learning in Robotics38 references4 citations

TL;DR

DOB-Net is a reinforcement learning framework that integrates a recurrent neural network-based disturbance observer with a controller to actively reject unknown, time-varying, and excessive disturbances in robotic systems. By encoding past and predicting future disturbances in the RNN hidden state, the method enables optimal control under actuator constraints, significantly outperforming conventional feedback controllers and classical RL algorithms in simulation and real-world tests with collected underwater disturbances.

ABSTRACT

This paper presents an observer-integrated Reinforcement Learning (RL) approach, called Disturbance OBserver Network (DOB-Net), for robots operating in environments where disturbances are unknown and time-varying, and may frequently exceed robot control capabilities. The DOB-Net integrates a disturbance dynamics observer network and a controller network. Originated from conventional DOB mechanisms, the observer is built and enhanced via Recurrent Neural Networks (RNNs), encoding estimation of past values and prediction of future values of unknown disturbances in RNN hidden state. Such encoding allows the controller generate optimal control signals to actively reject disturbances, under the constraints of robot control capabilities. The observer and the controller are jointly learned within policy optimization by advantage actor critic. Numerical simulations on position regulation tasks have demonstrated that the proposed DOB-Net significantly outperforms a conventional feedback controller and classical RL algorithms.

Motivation & Objective

• To address the challenge of controlling robots under unknown, excessive, and time-varying disturbances that exceed control capabilities.
• To overcome limitations of conventional disturbance observer-based control (DOBC), which rely on accurate system models and neglect time correlation and control constraints.
• To develop an integrated learning framework that jointly optimizes disturbance estimation and control policy for improved robustness and performance.
• To enable active disturbance rejection using predictive disturbance encoding via RNNs, enhancing adaptability to rapidly varying disturbances.
• To validate the method on both simulated disturbances and real-world data collected from an AUV in a water tank.

Proposed method

• DOB-Net combines a disturbance dynamics observer network and a controller network trained end-to-end via advantage actor-critic policy optimization.
• The observer network uses a recurrent neural network (RNN) to encode past disturbance values and predict future disturbances in its hidden state, mimicking and enhancing conventional DOB mechanisms.
• The RNN-based observer estimates and predicts disturbances without requiring an accurate system model, improving robustness to model uncertainties.
• Control signals are generated by the controller network using both the current robot state and the RNN-embedded disturbance prediction as inputs.
• The joint learning of observer and controller enables optimized representation of disturbance dynamics, avoiding reliance on hand-designed features.
• Training uses simulated disturbances composed of multiple sinusoidal waves; testing is performed on both simulated and real-world collected disturbances from an AUV's IMU.

Experimental results

Research questions

• RQ1 Can an RNN-enhanced observer network improve disturbance estimation and prediction for robots facing excessive, time-varying disturbances?
• RQ2 Can joint learning of observer and controller via RL achieve better disturbance rejection than classical feedback or standalone RL methods?
• RQ3 How does the DOB-Net perform on real-world, complex, and diverse disturbances not seen during training?
• RQ4 Does the method maintain performance under control constraints when disturbances exceed actuator capabilities?
• RQ5 Can the framework generalize to unseen disturbance patterns with wider amplitude ranges using limited training data?

Key findings

• DOB-Net achieved a median distance from target of 0.493m under large simulated disturbances, significantly narrowing the converged region compared to conventional controllers and classical RL.
• The ratio of median distances between DOB-Net and the optimal trajectory was 186.65% for large simulated disturbances, indicating strong performance close to the theoretical optimum.
• On collected real-world disturbances, the DOB-Net’s median distance was 264.88% of the optimal, showing a performance gap due to higher complexity and diversity in real data.
• Despite the gap on real data, DOB-Net still outperformed all baselines, including RISE and DOBC, demonstrating practical effectiveness in real-world scenarios.
• Larger disturbance amplitudes led to more similar optimal control patterns, making it easier for RL to learn near-optimal policies, which improved relative performance.
• The method showed robustness to model uncertainties and rapidly varying disturbances due to the RNN’s ability to encode temporal dynamics in the hidden state.