[Paper Review] A Narration-based Reward Shaping Approach using Grounded Natural Language Commands

Nicholas R. Waytowich, Sean L. Barton+2 more|arXiv (Cornell University)|Oct 31, 2019

Topic Modeling24 references4 citations

TL;DR

This paper proposes a narration-based reward shaping method that uses grounded natural language commands to improve deep reinforcement learning in sparse-reward environments like StarCraft II's BuildMarines mini-game. By projecting natural language commands and goal states into a shared embedding space via a mutual embedding model (MEM), the approach enables faster learning and generalization to unseen commands, outperforming both unshaped and traditional shaping baselines.

ABSTRACT

While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longstanding problem of reward sparsity. This is especially true for tasks such as training an agent to play StarCraft II, a real-time strategy game where reward is only given at the end of a game which is usually very long. While this problem can be addressed through reward shaping, such approaches typically require a human expert with specialized knowledge. Inspired by the vision of enabling reward shaping through the more-accessible paradigm of natural-language narration, we develop a technique that can provide the benefits of reward shaping using natural language commands. Our narration-guided RL agent projects sequences of natural-language commands into the same high-dimensional representation space as corresponding goal states. We show that we can get improved performance with our method compared to traditional reward-shaping approaches. Additionally, we demonstrate the ability of our method to generalize to unseen natural-language commands.

Motivation & Objective

• To address the challenge of sparse rewards in deep reinforcement learning, especially in complex, long-horizon environments like StarCraft II.
• To enable reward shaping through natural language narration, making it accessible to non-experts without low-level programming knowledge.
• To develop a method that generalizes to unseen natural language commands without retraining the embedding model.
• To evaluate whether natural-language-based reward shaping can outperform traditional shaping and baseline RL in terms of learning speed and final performance.

Proposed method

• The method uses a mutual embedding model (MEM) to map natural language commands and corresponding goal states into a shared high-dimensional representation space.
• Natural language commands are embedded using Word2Vec, while goal states are encoded from the agent's observation space using a neural network.
• The MEM learns a joint embedding space where semantically similar commands and goal states are mapped close to each other in vector space.
• A shaped reward is computed as the cosine similarity between the embedded command and the current state embedding, providing dense, intermediate feedback during training.
• The RL agent is trained using this auxiliary reward signal alongside the original sparse reward, improving learning efficiency.
• Generalization is evaluated by testing the trained MEM on novel, unseen natural language commands during RL training.

Experimental results

Research questions

• RQ1 Does narration-based reward shaping using grounded natural language commands improve learning performance compared to traditional reward shaping and unshaped baselines?
• RQ2 To what extent can the mutual embedding model generalize to novel, unseen natural language commands without retraining?
• RQ3 How does the performance of the RL agent degrade when using novel language commands, and can it still achieve meaningful task completion?
• RQ4 How does the semantic ambiguity of alternative language formulations affect the accuracy of command-state grounding and subsequent RL learning?

Key findings

• The narration-guided RL agent achieved significantly better performance than the unshaped baseline, with an average score of around 25–30 when using novel commands.
• The mutual embedding model successfully projected unseen natural language commands into the correct manifold of goal states, demonstrating semantic generalization.
• Commands with higher semantic ambiguity, such as 'choose a worker unit', showed reduced projection precision, leading to performance degradation.
• Despite reduced performance on novel commands, the narration-guided agent still outperformed the unshaped baseline, indicating residual benefit from the shaped reward.
• The t-SNE visualization confirmed that most novel commands projected near their original counterparts, but some deviated further, correlating with lower learning efficiency.
• The approach generalizes to diverse language formulations, suggesting potential for broader human-AI interaction in complex task environments.