[Paper Review] The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning

Vindula Jayawardana, Catherine So–kum Tang+3 more|arXiv (Cornell University)|Oct 16, 2022

Traffic control and management4 citations

TL;DR

This paper identifies the 'task underspecification problem' in deep reinforcement learning (DRL) evaluations, where relying on a few point MDPs instead of parameterized MDP families leads to misleading performance rankings. By evaluating DRL methods across families of MDPs—such as in cartpole, pendulum, and traffic signal control—the study shows that method rankings shift significantly depending on the MDP distribution, undermining empirical rigor and challenging the validity of state-of-the-art claims.

ABSTRACT

Evaluations of Deep Reinforcement Learning (DRL) methods are an integral part of scientific progress of the field. Beyond designing DRL methods for general intelligence, designing task-specific methods is becoming increasingly prominent for real-world applications. In these settings, the standard evaluation practice involves using a few instances of Markov Decision Processes (MDPs) to represent the task. However, many tasks induce a large family of MDPs owing to variations in the underlying environment, particularly in real-world contexts. For example, in traffic signal control, variations may stem from intersection geometries and traffic flow levels. The select MDP instances may thus inadvertently cause overfitting, lacking the statistical power to draw conclusions about the method's true performance across the family. In this article, we augment DRL evaluations to consider parameterized families of MDPs. We show that in comparison to evaluating DRL methods on select MDP instances, evaluating the MDP family often yields a substantially different relative ranking of methods, casting doubt on what methods should be considered state-of-the-art. We validate this phenomenon in standard control benchmarks and the real-world application of traffic signal control. At the same time, we show that accurately evaluating on an MDP family is nontrivial. Overall, this work identifies new challenges for empirical rigor in reinforcement learning, especially as the outcomes of DRL trickle into downstream decision-making.

Motivation & Objective

• To investigate how the arbitrary selection of a few point MDPs in DRL evaluations can lead to biased conclusions about method performance.
• To demonstrate that evaluating DRL methods on a family of MDPs—rather than isolated instances—can drastically alter the relative ranking of methods.
• To highlight the risk of evaluation overfitting when methods are optimized for specific MDP instances instead of the broader task family.
• To call for improved empirical rigor in DRL evaluations, especially as RL is deployed in real-world applications with high-stakes decision-making.
• To provide evidence that current benchmark practices may not reflect true algorithmic generalization within tasks, urging adoption of MDP family-based evaluation

Proposed method

• Augment standard DRL evaluations by modeling tasks as parameterized families of MDPs, where environment parameters (e.g., intersection geometry, traffic flow, number of lanes) define variations in the task.
• Generate diverse MDP instances by sampling from these parameterized families, such as 576 MDPs for cartpole and 180 for pendulum in the CARL benchmark suite.
• Evaluate multiple DRL algorithms (PPO, TRPO, SAC, TD3) across different random distributions of point MDPs within each family to assess ranking stability.
• Use normalized performance scores to compare methods across distributions and quantify ranking shifts across different MDP configurations.
• Conduct ablation studies on real-world traffic signal control and standard control benchmarks to validate the impact of MDP family evaluation.
• Analyze the impact of point MDP selection on cross-task generalization evaluations by combining one MDP from each task and computing ranking frequencies across all combinations.

Experimental results

Research questions

• RQ1 How does the choice of point MDPs affect the relative performance ranking of DRL algorithms within a single task?
• RQ2 To what extent does evaluating DRL methods on a family of MDPs change the conclusions about which method is state-of-the-art compared to point MDP evaluation?
• RQ3 Does the selection of MDP instances in standard benchmarks (e.g., pendulum, half-cheetah, mountain car) significantly alter the perceived robustness and generalization of DRL methods?
• RQ4 Can the use of MDP families improve the reliability and statistical power of DRL evaluations in real-world applications like traffic signal control?
• RQ5 How sensitive are DRL method rankings to variations in environment parameters such as traffic inflow, intersection geometry, or control parameters?

Key findings

• In the cartpole task, PPO was ranked first under distribution $D_1$ but ranked last under $D_5$, while TRPO outperformed PPO in the latter, demonstrating a complete reversal in ranking based on MDP distribution.
• In the pendulum task, similar ranking shifts were observed across different MDP distributions, indicating that method performance is highly sensitive to environment parameterization.
• When evaluating across tasks (pendulum, half-cheetah, mountain car), SAC was ranked first in 42.56% of MDP combination trials, while PPO was only first in 18.75% of cases, showing that default benchmarks may misrepresent method superiority.
• TD3 was ranked last in 47.85% of trials when evaluated across MDP combinations, indicating that its performance is highly unstable across different environment configurations.
• The study found that the choice of point MDPs used in standard benchmarks significantly affects method rankings, suggesting that current evaluation practices may not reflect true generalization capability.
• The results imply that relying on a single or small set of MDP instances for evaluation leads to insufficient statistical power and risks overfitting to specific configurations, undermining empirical rigor in DRL research.