[Paper Review] Learning high-level visual representations from a child's perspective without strong inductive biases

A. Emin Orhan, Brenden M. Lake|arXiv (Cornell University)|May 24, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper investigates whether high-level visual representations can be learned from a child’s egocentric visual experience without strong inductive biases. Using self-supervised learning on 200 hours of head-mounted camera video from a single child, the authors train vision transformers to learn robust semantic understanding and generative capabilities—achieving 70% of the performance of ImageNet-pretrained models in downstream tasks despite vastly different training data.

ABSTRACT

Young children develop sophisticated internal models of the world based on their visual experience. Can such models be learned from a child's visual experience without strong inductive biases? To investigate this, we train state-of-the-art neural networks on a realistic proxy of a child's visual experience without any explicit supervision or domain-specific inductive biases. Specifically, we train both embedding models and generative models on 200 hours of headcam video from a single child collected over two years and comprehensively evaluate their performance in downstream tasks using various reference models as yardsticks. On average, the best embedding models perform at a respectable 70% of a high-performance ImageNet-trained model, despite substantial differences in training data. They also learn broad semantic categories and object localization capabilities without explicit supervision, but they are less object-centric than models trained on all of ImageNet. Generative models trained with the same data successfully extrapolate simple properties of partially masked objects, like their rough outline, texture, color, or orientation, but struggle with finer object details. We replicate our experiments with two other children and find remarkably consistent results. Broadly useful high-level visual representations are thus robustly learnable from a representative sample of a child's visual experience without strong inductive biases.

Motivation & Objective

• To investigate whether high-level visual representations can be learned from a child’s real-world visual experience without strong inductive biases.
• To evaluate the capabilities of self-supervised learning models trained on longitudinal, egocentric video data from a single child.
• To assess how well such models generalize to downstream tasks like classification, segmentation, and generation compared to standard pre-trained models.
• To explore the robustness and consistency of these representations across multiple children and model architectures.
• To bridge the data gap between child development and deep learning by using a realistic proxy of early visual experience.

Proposed method

• Training vision transformer (ViT) models using three self-supervised learning (SSL) algorithms: DINO, Mugs, and masked autoencoders (MAE).
• Using a longitudinal, egocentric video dataset (SAYCam) totaling ~200 hours of continuous head-mounted camera footage from one child.
• Evaluating embedding models via fine-tuning on 9 downstream tasks (classification and segmentation) with frozen features.
• Training autoregressive transformer-based generative models on discrete latent codes from a learned codebook, enabling image completion and generation.
• Applying standard SSL data augmentations (e.g., color jittering, random cropping) to improve generalization during training.
• Replicating experiments on two additional children to assess consistency and robustness of learned representations.

Experimental results

Research questions

• RQ1 Can high-level visual representations be learned from a child’s egocentric visual experience without strong inductive biases?
• RQ2 How do models trained on child-like visual data compare to ImageNet-pretrained models in downstream visual recognition tasks?
• RQ3 To what extent do such models learn broad semantic categories and object localization without explicit supervision?
• RQ4 Can generative models trained on child visual data extrapolate partial object properties like shape, color, or orientation?
• RQ5 How consistent are the learned representations across different children and training runs?

Key findings

• The best embedding models achieved an average of 70% of the performance of a high-performing ImageNet-trained model on downstream classification and segmentation tasks.
• Models learned broad semantic categories and basic object localization capabilities without explicit supervision, though they were less object-centric than ImageNet-trained models.
• Generative models successfully extrapolated coarse object properties such as rough outline, texture, color, and orientation from partial masks, but struggled with fine-grained details.
• Results were remarkably consistent across three different children, indicating robustness of the learned representations to individual variation in visual experience.
• Despite limited data (≈40 days of visual input) and lower image quality, models demonstrated meaningful visual understanding, suggesting that rich, continuous visual experience alone can support high-level representation learning.
• The study demonstrates that powerful visual representations can be learned from a child’s real-world visual experience using only generic deep learning architectures and self-supervised learning, without domain-specific inductive biases.