[Paper Review] TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

Gabriel Synnaeve, Nantas Nardelli+6 more|arXiv (Cornell University)|Nov 1, 2016

Artificial Intelligence in Games1 references84 citations

TL;DR

TorchCraft is a library that bridges Torch, a machine learning framework, with StarCraft: Brood War, enabling end-to-end reinforcement learning and imitation learning on real-time strategy games. It provides a client-server interface for real-time state exchange, supports both game-controlled and game-attached modes, and facilitates research on complex, partially observable, high-dimensional environments with structured state representations and replay data access.

ABSTRACT

We present TorchCraft, a library that enables deep learning research on Real-Time Strategy (RTS) games such as StarCraft: Brood War, by making it easier to control these games from a machine learning framework, here Torch. This white paper argues for using RTS games as a benchmark for AI research, and describes the design and components of TorchCraft.

Motivation & Objective

• To address the lack of accessible, scalable interfaces between machine learning frameworks and real-time strategy (RTS) games like StarCraft: Brood War.
• To enable researchers to train agents using deep reinforcement learning and imitation learning on high-dimensional, partially observable, and structurally complex game environments.
• To provide a standardized, efficient, and extensible interface that supports both real-time gameplay and replay analysis for training and evaluation.
• To support the integration of structured game state representations alongside raw pixel inputs, improving model interpretability and training efficiency.
• To foster research in complex decision-making by offering a production-ready bridge between Torch and StarCraft: Brood War, enabling reproducible experiments and benchmarking.

Proposed method

• Dynamically injects a server-side DLL into the StarCraft: Brood War game engine to establish a real-time client-server communication channel.
• Uses a synchronous, bidirectional protocol where the game engine sends state data to the machine learning client and receives actions via the Torch framework.
• Supports two execution modes: 'game-controlled' (reconnects between matches, enabling multiple learning instances) and 'game-attached' (persistent connection, no reconnection needed, but limited to one instance per OS).
• Exposes a simple API via Torch, including connect(), receive(), send(), and helper functions for StarCraft-specific state and action handling.
• Provides efficient storage and retrieval of game frames and replay data for offline analysis and imitation learning.
• Integrates with existing deep learning models by enabling feature extraction from structured game states and raw pixel inputs, supporting hybrid modeling approaches.

Experimental results

Research questions

• RQ1 Can a standardized, low-level interface between machine learning frameworks and RTS games like StarCraft: Brood War enable scalable and reproducible reinforcement learning research?
• RQ2 How can structured game state representations be effectively integrated with raw pixel inputs to improve learning efficiency and model interpretability in complex environments?
• RQ3 To what extent can a real-time, bidirectional communication pipeline between a game engine and a deep learning framework support end-to-end training of agents in partially observable, high-dimensional environments?
• RQ4 Can the library support both online reinforcement learning and offline imitation learning using human demonstration data and game replays?
• RQ5 How does the choice between game-controlled and game-attached execution modes affect the scalability and stability of multi-agent training in RTS environments?

Key findings

• TorchCraft successfully enables end-to-end deep reinforcement learning on StarCraft: Brood War by providing a stable, low-latency interface between Torch and the game engine.
• The library supports both real-time gameplay and replay-based training, allowing researchers to leverage existing human-play data and game traces for imitation learning.
• The dual execution modes—game-controlled and game-attached—offer flexibility in experimental design, balancing scalability and connection stability.
• The integration of structured state representations with raw pixel inputs allows for more efficient and targeted training, particularly when shaping loss functions or designing reward functions.
• TorchCraft has already been used in published reinforcement learning experiments on StarCraft, with results reported in [23] and planned for open-sourcing within the library.
• The library enables researchers to bypass the complexity of reverse-engineering game APIs, significantly lowering the barrier to entry for RTS AI research using deep learning frameworks.