[Paper Review] LPaintB: Learning to Paint from Self-Supervision

Biao Jia, Jonathan Brandt+3 more|arXiv (Cornell University)|Jun 17, 2019

Generative Adversarial Networks and Image Synthesis39 references4 citations

TL;DR

LPaintB proposes a reinforcement learning-based painting agent that learns to reproduce reference images using brush strokes by combining self-supervised learning with proximal policy optimization (PPO). It automatically generates positive training data by repurposing final states from failed rollouts, significantly improving sample efficiency and performance over training from scratch, achieving high-quality results in under 30 seconds on a GTX 1080 GPU for 1000×800 images with 20,000 strokes.

ABSTRACT

We present a novel reinforcement learning-based natural media painting algorithm. Our goal is to reproduce a reference image using brush strokes and we encode the objective through observations. Our formulation takes into account that the distribution of the reward in the action space is sparse and training a reinforcement learning algorithm from scratch can be difficult. We present an approach that combines self-supervised learning and reinforcement learning to effectively transfer negative samples into positive ones and change the reward distribution. We demonstrate the benefits of our painting agent to reproduce reference images with brush strokes. The training phase takes about one hour and the runtime algorithm takes about 30 seconds on a GTX1080 GPU reproducing a 1000x800 image with 20,000 strokes.

Motivation & Objective

• To develop an intelligent painting agent that can learn to reproduce reference images using only the target image and no paired human stroke data.
• To address the challenge of sparse reward signals in reinforcement learning for natural media painting by improving sample efficiency.
• To enable generalization to unseen styles and images without requiring expert-annotated action sequences.
• To reduce reliance on expensive human-annotated datasets by leveraging self-supervised data collection from failed rollouts.
• To accelerate training and improve policy performance through a hybrid self-supervised and reinforcement learning framework.

Proposed method

• The method uses proximal policy optimization (PPO) with observations encoding both the current rendering and the target reference image.
• It introduces a self-supervised data collection mechanism that repurposes the final state of failed rollouts as positive training examples, replacing the goal state with the final rendering.
• The action space is continuous and defined by brush parameters: position, orientation, length, and size.
• The training phase uses only reference images and no paired human stroke data, relying instead on self-supervised data generated from unsuccessful rollouts.
• The policy network is fine-tuned using both self-supervised data and reinforcement learning, enabling faster convergence and better performance.
• An efficient rendering system generates high-resolution stroke-based paintings in real-time using the trained agent.

Experimental results

Research questions

• RQ1 Can self-supervised learning effectively generate positive training data for reinforcement learning in natural media painting without human-annotated actions?
• RQ2 How does combining self-supervised data collection with reinforcement learning improve sample efficiency and policy performance compared to training from scratch?
• RQ3 To what extent can the trained agent generalize to unseen reference images and different artistic styles?
• RQ4 How does the proposed method compare in performance and efficiency to prior reinforcement learning-based painting agents trained from scratch?
• RQ5 What impact does the distribution of self-supervised data have on the generalization and visual quality of the generated paintings?

Key findings

• The combined self-supervised and reinforcement learning scheme achieved a cumulative reward of 61.13 on Benchmark2 (reproducing from a blank canvas), significantly outperforming reinforcement learning only (26.33) and self-supervised only (30.79).
• On high-resolution image reconstruction, LPaintB achieved a lower L₂ loss of 1485 compared to 1920 for PaintBot [JFB*19], indicating better image fidelity.
• The method reduced training time to approximately one hour and inference time to about 30 seconds on a GTX 1080 GPU for a 1000×800 image with 20,000 strokes.
• The learning curve showed faster convergence and higher performance when self-supervision was used, confirming improved sample efficiency.
• The approach successfully generated visually compelling paintings without any paired human stroke data, demonstrating effective zero-shot generalization to unseen images.
• Despite improvements, the generated images remain somewhat blurry in high-contrast regions, indicating a need for better reward shaping or higher-resolution supervision.