[Paper Review] Unsupervised Behaviour Discovery with Quality-Diversity Optimisation

Luca Grillotti, Antoine Cully|arXiv (Cornell University)|Jun 10, 2021

Evolutionary Algorithms and Applications8 citations

TL;DR

This paper proposes AURORA, a Quality-Diversity algorithm that autonomously learns behavioural descriptors from raw sensory data, eliminating the need for hand-coded descriptors. By combining unsupervised dimensionality reduction with QD optimisation, AURORA discovers diverse, high-performing robot behaviours comparable to hand-coded baselines while capturing novelty across more behavioural features.

ABSTRACT

Quality-Diversity algorithms refer to a class of evolutionary algorithms designed to find a collection of diverse and high-performing solutions to a given problem. In robotics, such algorithms can be used for generating a collection of controllers covering most of the possible behaviours of a robot. To do so, these algorithms associate a behavioural descriptor to each of these behaviours. Each behavioural descriptor is used for estimating the novelty of one behaviour compared to the others. In most existing algorithms, the behavioural descriptor needs to be hand-coded, thus requiring prior knowledge about the task to solve. In this paper, we introduce: Autonomous Robots Realising their Abilities, an algorithm that uses a dimensionality reduction technique to automatically learn behavioural descriptors based on raw sensory data. The performance of this algorithm is assessed on three robotic tasks in simulation. The experimental results show that it performs similarly to traditional hand-coded approaches without the requirement to provide any hand-coded behavioural descriptor. In the collection of diverse and high-performing solutions, it also manages to find behaviours that are novel with respect to more features than its hand-coded baselines. Finally, we introduce a variant of the algorithm which is robust to the dimensionality of the behavioural descriptor space.

Motivation & Objective

• To address the limitation of hand-coded behavioural descriptors in Quality-Diversity (QD) algorithms, which require prior task knowledge and may bias diversity.
• To enable robots to autonomously discover a repertoire of diverse and high-performing behaviours using only raw sensory data.
• To develop a QD algorithm that learns meaningful, low-dimensional behavioural descriptors in an unsupervised manner.
• To ensure robustness to the dimensionality of the learned behavioural descriptor space through a novel container size control mechanism.
• To evaluate the method on simulated robotic tasks and compare it to traditional hand-coded QD baselines.

Proposed method

• AURORA alternates between a QD optimisation phase and an encoder update phase to iteratively improve behavioural descriptor learning.
• Raw sensory data from robot controllers is encoded into low-dimensional behavioural descriptors using a neural autoencoder trained via contrastive learning.
• Novelty is computed based on the distance between encoded behavioural descriptors in the container, using k-nearest neighbours with an exclusive ε-dominance criterion.
• The container maintains diversity and performance by adding only individuals with high fitness and sufficient novelty relative to existing members.
• A Container Size Control (CSC) mechanism dynamically adjusts the container size using a proportional control loop to maintain a target size.
• The encoder is periodically retrained on sensory data from the current container to adapt to evolving behavioural distributions.

Experimental results

Research questions

• RQ1 Can a QD algorithm learn effective behavioural descriptors from raw sensory data without hand-coded specifications?
• RQ2 How does the performance and diversity of AURORA compare to traditional hand-coded QD baselines in terms of behavioural coverage and fitness?
• RQ3 To what extent does AURORA capture behavioural novelty across more features than hand-coded descriptors?
• RQ4 How robust is AURORA to variations in the dimensionality of the learned behavioural descriptor space?
• RQ5 Can AURORA generalise to complex robotic tasks where hand-coded descriptors are difficult to define?

Key findings

• AURORA achieves performance comparable to hand-coded QD baselines in three simulated robotic tasks: maze navigation, hexapod walking, and air hockey.
• The algorithm discovers behaviours that are novel with respect to more features than the hand-coded baselines, indicating broader behavioural diversity.
• AURORA-CSC, the variant with container size control, maintains stable container size and performance across different descriptor space dimensions.
• The performance of AURORA depends on the information content of the sensory data relative to the hand-coded BD; when sensory data is richer, AURORA captures more nuanced behavioural distinctions.
• The container size control mechanism effectively prevents container blow-up, even when the learned descriptor space dimensionality changes.
• The method shows promise for real-world deployment, though future work is needed to evaluate it in non-resetting, partially observable environments.