[Paper Review] Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

Mehdi Azabou, Michael Mendelson+5 more|arXiv (Cornell University)|Mar 15, 2023

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes a self-supervised multi-timescale representation learning framework, Bootstrap Across Multiple Scales (BAMS), that uses separate latent spaces for short- and long-term behavioral dynamics. By predicting future action distributions rather than sequences, and training via a latent predictive loss without negative views, BAMS achieves state-of-the-art performance on the MABe 2022 benchmark, ranking first overall and first on all sequence-level tasks.

ABSTRACT

Natural behavior consists of dynamics that are complex and unpredictable, especially when trying to predict many steps into the future. While some success has been found in building representations of behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings where behavior becomes increasingly hard to model. In this work, we develop a multi-task representation learning model for behavior that combines two novel components: (i) An action prediction objective that aims to predict the distribution of actions over future timesteps, and (ii) A multi-scale architecture that builds separate latent spaces to accommodate short- and long-term dynamics. After demonstrating the ability of the method to build representations of both local and global dynamics in realistic robots in varying environments and terrains, we apply our method to the MABe 2022 Multi-agent behavior challenge, where our model ranks 1st overall and on all global tasks, and 1st or 2nd on 7 out of 9 frame-level tasks. In all of these cases, we show that our model can build representations that capture the many different factors that drive behavior and solve a wide range of downstream tasks.

Motivation & Objective

• To address the challenge of modeling complex, multi-timescale behavioral dynamics in naturalistic settings where annotations are scarce.
• To develop a self-supervised representation learning method that captures both fine-grained short-term dynamics and global long-term behavioral patterns.
• To enable robust downstream task performance across diverse behavioral analysis tasks, including frame-level and sequence-level prediction.
• To overcome limitations of existing contrastive or reconstruction-based methods that focus too narrowly on local dynamics or fail to disentangle timescales.

Proposed method

• The method employs two separate temporal convolutional networks with distinct receptive fields to learn representations in separate short-term and long-term latent spaces.
• It introduces a novel action-prediction objective that predicts the distribution of actions over future timesteps, rather than predicting exact action sequences.
• The model uses a self-supervised, bootstrap-based training scheme that relies only on positive views within each timescale, avoiding the need for negative samples.
• A predictive regression loss is applied to align the predicted action distribution with the ground-truth future action distribution in the representation space.
• The architecture enables processing of long sequences by decoupling timescale-specific dynamics and preserving granularity in each space.
• The method is trained end-to-end using a contrastive-like objective within each timescale, but without requiring negative pairs, enabling scalability to long sequences.

Experimental results

Research questions

• RQ1 Can a self-supervised model learn disentangled representations of short-term and long-term behavioral dynamics without explicit supervision?
• RQ2 Does predicting future action distributions rather than sequences improve modeling of complex, multi-timescale behavior?
• RQ3 Can the model generalize across diverse behavioral tasks, including both frame-level and sequence-level prediction, without task-specific fine-tuning?
• RQ4 How does the separation of latent spaces for different timescales affect performance on downstream tasks involving global behavior patterns?

Key findings

• On the MABe 2022 Multi-agent Behavior Challenge, the model achieved first place overall, ranking first on all 4 sequence-level tasks and within the top 3 on all 9 frame-level tasks.
• The model achieved a 10% performance gap over the next best method on the sequence-level task of decoding mouse strain, demonstrating strong global representation learning.
• In synthetic quadruped robot environments, the model successfully captured both robot morphological properties and terrain dynamics without explicit supervision.
• The method outperformed existing contrastive and reconstruction-based approaches by maintaining fine-grained temporal dynamics while also modeling long-term behavioral structure.
• The absence of negative samples in training allowed the model to process longer sequences than typical contrastive methods, enhancing scalability.