[Paper Review] Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts

Yingtao Tian, David Ha|arXiv (Cornell University)|Sep 18, 2021

3D Surveying and Cultural Heritage4 citations

TL;DR

This paper proposes using modern evolution strategies (ES) to generate geometric abstract art by optimizing the placement and color of transparent triangles to match either target images or text prompts via CLIP embeddings. It demonstrates that ES achieves quality and efficiency comparable to gradient-based methods while producing diverse, human-aligned abstractions distinct from differentiable rendering, especially in abstract expressionist styles.

ABSTRACT

Evolutionary algorithms have been used in the digital art scene since the 1970s. A popular application of genetic algorithms is to optimize the procedural placement of vector graphic primitives to resemble a given painting. In recent years, deep learning-based approaches have also been proposed to generate procedural drawings, which can be optimized using gradient descent. In this work, we revisit the use of evolutionary algorithms for computational creativity. We find that modern evolution strategies (ES) algorithms, when tasked with the placement of shapes, offer large improvements in both quality and efficiency compared to traditional genetic algorithms, and even comparable to gradient-based methods. We demonstrate that ES is also well suited at optimizing the placement of shapes to fit the CLIP model, and can produce diverse, distinct geometric abstractions that are aligned with human interpretation of language. Videos and demo: https://es-clip.github.io/

Motivation & Objective

• To revisit evolutionary algorithms for computational creativity using modern evolution strategies (ES) as an alternative to gradient-based optimization.
• To improve upon traditional genetic algorithms in quality and efficiency when synthesizing images using geometric primitives.
• To enable alignment with human-interpretation of language by optimizing shape placements using CLIP embeddings for text prompts.
• To explore whether ES can produce diverse, distinct geometric abstractions that reflect abstract concepts, not just concrete images.
• To demonstrate that ES offers a domain-agnostic, flexible alternative to gradient-based methods in procedural art generation.

Proposed method

• The method uses evolution strategies (ES) to optimize the positions, colors, and opacities of 50 transparent triangles on a canvas to minimize the cosine distance between the rendered image and a target image or text prompt.
• Fitness is computed using CLIP’s image and text encoders, where the similarity between the rendered canvas and the CLIP-embedded target is used as the objective function.
• The optimization employs the PGPE (Policy Gradient with Parameter Exponential) variant of ES with ClipUp, enabling efficient, non-differentiable rendering optimization.
• The rendering process is implemented using nvdiffrast for differentiable triangle composition, allowing gradient-based comparison while the main optimization remains ES-based.
• Multiple independent runs are performed per prompt to assess diversity and robustness of the generated abstractions.
• The method supports both image fitting (e.g., Mona Lisa) and text-to-image generation (e.g., 'a painting of a mountain in the style of Kandinsky').

Experimental results

Research questions

• RQ1 Can modern evolution strategies outperform traditional genetic algorithms in quality and efficiency for procedural image synthesis using geometric primitives?
• RQ2 How does ES-based optimization compare to gradient-based methods in generating abstract art aligned with human-interpretation of language?
• RQ3 To what extent can ES produce diverse, distinct geometric abstractions from the same text prompt, reflecting semantic concepts?
• RQ4 What artistic styles emerge from ES optimization, and how do they differ from those produced by differentiable rendering?
• RQ5 Can ES serve as a domain-agnostic alternative to gradient-based methods in procedural art generation?

Key findings

• Modern evolution strategies (ES) achieve image synthesis quality and optimization efficiency comparable to state-of-the-art differentiable rendering methods, despite being non-differentiable.
• The ES-based method produces geometric abstractions that are clearly aligned with human interpretation of text prompts, such as 'a mountain' or 'a city at night', with high semantic fidelity.
• Multiple independent runs on the same text prompt yield diverse, distinct results, demonstrating strong exploration and creative variability.
• The ES-generated art exhibits a bolder, more abstract expressionist style with clearer boundaries and geometric forms, contrasting with the softer, more blended results from gradient-based optimization.
• The method produces unique artistic outcomes not seen in gradient-based approaches, suggesting intrinsic differences in optimization dynamics.
• The approach is robust and generalizable, as it works equally well for both concrete image fitting and abstract concept generation using CLIP embeddings.