[Paper Review] Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge

Keerthiram Murugesan, Mattia Atzeni+4 more|arXiv (Cornell University)|May 2, 2020

Topic Modeling33 references17 citations

TL;DR

This paper proposes a reinforcement learning agent for text-based environments that integrates symbolic belief graphs with commonsense knowledge from ConceptNet to reduce exploration and improve sample efficiency. By dynamically fusing real-time environmental observations with external knowledge, the agent achieves superior performance on kitchen cleanup and cooking recipe tasks, though excessive knowledge can hinder learning, highlighting the need for context-aware knowledge integration.

ABSTRACT

In this paper, we consider the recent trend of evaluating progress on reinforcement learning technology by using text-based environments and games as evaluation environments. This reliance on text brings advances in natural language processing into the ambit of these agents, with a recurring thread being the use of external knowledge to mimic and better human-level performance. We present one such instantiation of agents that use commonsense knowledge from ConceptNet to show promising performance on two text-based environments.

Motivation & Objective

• To address the sample inefficiency of text-based reinforcement learning agents by incorporating commonsense knowledge.
• To investigate how external knowledge from ConceptNet can reduce exploration in text-based environments like TextWorld.
• To explore the conditions under which commonsense knowledge improves or harms agent performance.
• To develop a dual-graph architecture that symbolically represents both the agent's current belief state and global commonsense knowledge.
• To evaluate the method on diverse text-based tasks, including kitchen cleanup and recipe-based games.

Proposed method

• The agent maintains a local belief graph that symbolically represents its current perception of the environment based on textual observations.
• It constructs a global commonsense graph using entities and relations from ConceptNet to represent external world knowledge.
• The agent fuses the belief graph and commonsense graph through a differentiable attention mechanism to guide action selection.
• The system supports two knowledge integration modes: full-graph (all knowledge available from start) and evolve-graph (knowledge incrementally added as the agent explores).
• The approach is evaluated using a Graph-Aided Transformer Agent (GATA)-inspired framework, with state representation learned via self-attention over symbolic graphs.
• Performance is measured across multiple runs on two text-based tasks: kitchen cleanup and recipe-based cooking games.

Experimental results

Research questions

• RQ1 Can integrating commonsense knowledge from ConceptNet reduce exploration and improve sample efficiency in text-based RL environments?
• RQ2 How does the timing of knowledge integration (full vs. incremental) affect agent performance?
• RQ3 Under what conditions does external knowledge degrade agent performance rather than improve it?
• RQ4 How does the fusion of symbolic belief graphs and commonsense knowledge enhance decision-making in text-based games?
• RQ5 In what scenarios is ground-truth belief graph information more beneficial than commonsense knowledge?

Key findings

• The proposed agent outperformed both the simple text-based agent and the GATA baseline on the kitchen cleanup task, demonstrating reduced exploration and faster convergence.
• On the cooking recipe task, the evolve-graph setting (incremental knowledge) outperformed the full-graph setting, indicating that delayed knowledge integration reduces noise and improves performance.
• The GATA_Full agent performed best on the cooking recipe task, suggesting that in simple tasks with localized goals, ground-truth state information is more effective than commonsense knowledge.
• Excessive or poorly filtered commonsense knowledge can overwhelm the agent, leading to degraded performance, especially in low-complexity environments.
• The dual-graph approach (belief + commonsense) significantly improved performance on complex tasks like kitchen cleanup, where reasoning about object locations and relationships was essential.
• The results show that knowledge integration must be context-aware: the utility of commonsense knowledge depends on task complexity and environmental structure.