[Paper Review] Learning When to Look: On-Demand Keypoint-Video Fusion for Animal Behavior Analysis

Weihan Li, Jingyang Ke+3 more|arXiv (Cornell University)|Mar 7, 2026

Zebrafish Biomedical Research Applications0 citations

TL;DR

LookAgain combines keypoint-based motion with on-demand visual grounding to analyze animal behavior, training a gating module to activate video processing only when keypoint signals are ambiguous, achieving comparable performance with far fewer frames.

ABSTRACT

Understanding animal behavior from video is essential for neuroscience research. Modern laboratories typically collect two complementary data streams: skeletal keypoints from pose estimation tools and raw video recordings. Keypoint-based methods are efficient but suffer from geometric ambiguity, environmental blindness, and sensitivity to occlusions. Video-based methods capture rich context but require processing every frame, making them impractical for the hundreds of hours of recordings that modern experiments produce. We introduce LookAgain, a multimodal framework that combines the efficiency of keypoints with the representational power of video through on-demand visual grounding. During training, LookAgain uses dense visual features to pretrain a motion encoder and to train a gating module that learns which frames require visual context. During inference, this gating module activates visual processing only when keypoint signals are ambiguous, while maintaining performance comparable to using all frames. Experiments on single-animal and multi-animal benchmarks show that LookAgain achieves strong performance with significantly reduced computational cost, enabling high-quality behavior analysis on long-duration recordings.

Motivation & Objective

• Motivate efficient animal behavior analysis by leveraging both keypoints and video.
• Develop a training-inference decoupled framework to use dense visual features for pretraining and a learned gate for on-demand inference.
• Enable both supervised behavior classification and unsupervised behavior segmentation on long-duration recordings.

Proposed method

• Pretrain a Motion Encoder on keypoint sequences with dense visual guidance using four losses (mask, cross-modal vision prediction, motion reconstruction, RVQ commitment).
• Tokenize motion with Residual Vector Quantization and model temporal context with a Transformer; model social context with cross-animal attention when multiple animals are present.
• Extract visual features from a frozen vision encoder and align them to the motion space through a visual projection module.
• Fine-tune with a gating module that selects top-k informative frames to activate visual processing, enabling on-demand fusion of motion and visual features.
• Fuse gated visual representations with motion embeddings for supervised classification and unsupervised segmentation, using DEC-based clustering for segmentation and a total-variation penalty to encourage temporal coherence.

Experimental results

Research questions

• RQ1 Can an on-demand visual grounding gate reduce computation while maintaining performance in animal behavior analysis?
• RQ2 Does training-inference decoupling with dense visual pretraining improve downstream classification and segmentation under label-scarce conditions?
• RQ3 How does keypoint-video fusion perform in single-animal vs. multi-animal social behaviors, especially under occlusion or identity ambiguity?
• RQ4 What is the impact of gating components (keypoint reliability, motion saliency, semantic context) on gating effectiveness?

Key findings

• Activating only 25% of frames with on-demand visual grounding achieves comparable performance to using all frames in supervised tasks.
• Pretraining the Motion Encoder with dense visual features improves downstream supervised and unsupervised performance over training from scratch.
• The gating mechanism outperforms uniform frame sampling, with motion saliency (q_t) being the most important component.
• Keypoint-video fusion yields strongest results across multiple social behaviors, though some behaviors like Oral Contact may rely more on precise geometric keypoints.
• Unsupervised segmentation benefits from visual grounding and can discover interpretable behaviors like Feeding that keypoints alone miss.