[Paper Review] Language is Power: Representing States Using Natural Language in Reinforcement Learning

Erez Schwartz, Guy Tennenholtz+2 more|arXiv (Cornell University)|Oct 2, 2019

Reinforcement Learning in Robotics26 references4 citations

TL;DR

This paper proposes representing states in reinforcement learning using natural language instead of raw visual inputs, leveraging pre-trained language models to encode semantic meaning. In ViZDoom environments, language-based agents outperform vision-based counterparts in sample efficiency, robustness, and final performance, demonstrating that natural language is a powerful semantic state representation for RL.

ABSTRACT

Recent advances in reinforcement learning have shown its potential to tackle complex real-life tasks. However, as the dimensionality of the task increases, reinforcement learning methods tend to struggle. To overcome this, we explore methods for representing the semantic information embedded in the state. While previous methods focused on information in its raw form (e.g., raw visual input), we propose to represent the state using natural language. Language can represent complex scenarios and concepts, making it a favorable candidate for representation. Empirical evidence, within the domain of ViZDoom, suggests that natural language based agents are more robust, converge faster and perform better than vision based agents, showing the benefit of using natural language representations for reinforcement learning.

Motivation & Objective

• To investigate whether natural language can serve as an effective semantic state representation in reinforcement learning.
• To compare language-based state representations with raw visual and semantic segmentation representations in complex RL environments.
• To evaluate the robustness and sample efficiency of language-based agents under environmental perturbations and nuisance factors.
• To explore the feasibility of using natural language for encoding subjective, unstructured, and high-dimensional state information.
• To assess the potential of language representations for improving interpretability and accountability in RL agents.

Proposed method

• The state is represented using natural language descriptions generated via a rule-based or learned parser, capturing semantic content of the environment.
• Sentence embeddings are extracted using pre-trained contextualized language models (e.g., BERT), followed by a 1D convolutional neural network for feature extraction.
• The resulting sentence representations are used as input to standard deep RL algorithms, including DQN and PPO, for policy learning.
• The framework is evaluated in the ViZDoom environment, where states are described using natural language sentences reflecting spatial, object, and action-related information.
• Robustness is tested by introducing nuisance factors such as visual clutter and environmental changes, measuring performance degradation across representation types.
• Hyperparameter ablation studies are conducted to analyze the impact of input sequence length and number of patches in the language model.

Experimental results

Research questions

• RQ1 Can natural language representations improve sample efficiency and final performance in deep reinforcement learning compared to raw visual inputs?
• RQ2 How does the robustness of language-based agents compare to vision-based agents under environmental perturbations and nuisance factors?
• RQ3 In what types of environments or state spaces is natural language representation particularly advantageous?
• RQ4 Can natural language representations effectively encode subjective, transient, or unstructured information that is difficult to represent in pixel or vector form?
• RQ5 To what extent can language-based state representations enhance interpretability and accountability in RL agents?

Key findings

• Language-based agents achieved higher final average rewards than vision-based agents in the ViZDoom 'Defend the Center' scenario, demonstrating superior performance.
• Language-based agents converged faster and exhibited greater sample efficiency, reducing the number of training steps required to reach peak performance.
• Robustness evaluations showed that language-based agents were significantly more resilient to environmental nuisances and visual clutter than vision-based agents.
• The performance of language-based agents remained stable even when the environment introduced distracting elements, indicating stronger generalization capabilities.
• Ablation studies revealed that increasing the number of input patches in the language model improved performance, suggesting that richer textual context enhances representation quality.
• The study demonstrates that natural language representations can effectively encode complex semantic content, including spatial relations, object states, and task-relevant dynamics, outperforming raw pixel inputs in semantic richness and learning efficiency.