[Paper Review] Learning to Play General Video-Games via an Object Embedding Network

William Woof, Ke Chen|arXiv (Cornell University)|Mar 14, 2018

Reinforcement Learning in Robotics18 references3 citations

TL;DR

This paper proposes an object embedding network (OEN) that compresses variable-length object feature vectors from video games into a fixed-length unified representation, enabling deep reinforcement learning (DRL) agents to learn directly from structured game-state objects. Evaluated on five GVG-AI games, the OEN-based agent achieves performance comparable to state-of-the-art DRL agents using raw pixel inputs, demonstrating the viability of object-based representation for general video-game AI.

ABSTRACT

Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However most current DRL video-game agents learn end-to-end from the video-output of the game, which is superfluous for many applications and creates a number of additional problems. More importantly, directly working on pixel-based raw video data is substantially distinct from what a human player does.In this paper, we present a novel method which enables DRL agents to learn directly from object information. This is obtained via use of an object embedding network (OEN) that compresses a set of object feature vectors of different lengths into a single fixed-length unified feature vector representing the current game-state and fulfills the DRL simultaneously. We evaluate our OEN-based DRL agent by comparing to several state-of-the-art approaches on a selection of games from the GVG-AI Competition. Experimental results suggest that our object-based DRL agent yields performance comparable to that of those approaches used in our comparative study.

Motivation & Objective

• Address the limitations of end-to-end deep reinforcement learning (DRL) agents that rely solely on raw pixel inputs from video games.
• Overcome issues such as high computational cost, poor generalization, and lack of interpretability inherent in pixel-based representations.
• Enable DRL agents to learn from semantically meaningful, game-internal object data that better reflects human-like game understanding.
• Develop a unified, fixed-length representation of dynamic game states composed of variable numbers of objects with diverse attributes.
• Demonstrate that object-based DRL can achieve performance on par with pixel-based SOTA agents across diverse video games.

Proposed method

• Extract object-level features (e.g., position, velocity, type) from the game state using game engine access or semantic segmentation of video.
• Represent each object as a fixed-length feature vector containing its class and attributes, ensuring consistency across object types.
• Use a set-based neural network architecture (OEN) to process a variable-sized set of object vectors and produce a single fixed-length embedding vector.
• Train the OEN end-to-end using a DRL algorithm (e.g., DQN or Rainbow) where the agent's policy is conditioned on the OEN-embedded game state.
• Apply attention mechanisms or permutation-equivariant layers in the OEN to handle object order invariance and capture relational context.
• Optimize the OEN and policy jointly via reinforcement learning using sparse rewards from game scores.

Experimental results

Research questions

• RQ1 Can a DRL agent trained on object-based representations outperform or match agents trained on raw pixel data in general video-game playing?
• RQ2 How effective is a learnable object embedding network (OEN) at compressing variable-length, heterogeneous object sets into a fixed-length game-state representation?
• RQ3 Does object-based representation reduce training time and improve sample efficiency compared to pixel-based DRL?
• RQ4 How does the performance of the OEN-based agent vary across different game types and levels of game-state complexity?
• RQ5 To what extent can object-based DRL generalize across games without retraining or architectural changes?

Key findings

• The OEN-based DRL agent successfully learned to play all five GVG-AI competition games tested, demonstrating robustness across diverse game mechanics.
• Performance of the OEN agent was comparable to both pixel-based DRL agents and handcrafted feature baselines across all five games.
• The handcrafted feature baseline outperformed the OEN agent in some games, suggesting that certain game-specific features remain highly effective.
• Despite training for only two million steps (equivalent to ~8 million frames), the OEN agent achieved stable learning curves and consistent performance across multiple runs.
• The OEN model effectively compressed variable-length object inputs into a fixed-length representation, enabling compatibility with standard DRL architectures.
• The method showed promise in handling games with dynamic object counts and complex interactions, suggesting scalability to richer game environments.