[Paper Review] From Third Person to First Person: Dataset and Baselines for Synthesis and Retrieval

Mohamed Elfeki, Krishna Regmi+2 more|arXiv (Cornell University)|Dec 1, 2018

Generative Adversarial Networks and Image Synthesis37 references17 citations

TL;DR

This paper introduces a novel synthetic and real-world dataset of synchronized egocentric (first-person) and exocentric (third-person) video pairs, and proposes a conditional GAN-based framework for first-person image synthesis from third-person views. It demonstrates that domain adaptation from synthetic to real data improves cross-view retrieval performance, achieving 30.82% top-1 accuracy on real egocentric data using a two-stream CNN with domain adaptation.

ABSTRACT

First-person (egocentric) and third person (exocentric) videos are drastically different in nature. The relationship between these two views have been studied in recent years, however, it has yet to be fully explored. In this work, we introduce two datasets (synthetic and natural/real) containing simultaneously recorded egocentric and exocentric videos. We also explore relating the two domains (egocentric and exocentric) in two aspects. First, we synthesize images in the egocentric domain from the exocentric domain using a conditional generative adversarial network (cGAN). We show that with enough training data, our network is capable of hallucinating how the world would look like from an egocentric perspective, given an exocentric video. Second, we address the cross-view retrieval problem across the two views. Given an egocentric query frame (or its momentary optical flow), we retrieve its corresponding exocentric frame (or optical flow) from a gallery set. We show that using synthetic data could be beneficial in retrieving real data. We show that performing domain adaptation from the synthetic domain to the natural/real domain, is helpful in tasks such as retrieval. We believe that the presented datasets and the proposed baselines offer new opportunities for further research in this direction. The code and dataset are publicly available.

Motivation & Objective

• To bridge the gap between egocentric (first-person) and exocentric (third-person) video domains by creating a new dataset with synchronized recordings.
• To enable first-person video synthesis from third-person views using conditional GANs, despite drastic differences in field of view and perspective.
• To improve cross-view retrieval performance by leveraging synthetic data for domain adaptation to real egocentric data.
• To evaluate the effectiveness of view-invariant representations learned across egocentric and exocentric domains for retrieval and action recognition.

Proposed method

• Collect a large-scale synthetic dataset using game engines with frame-level annotations for egocentric/exocentric camera poses and human actions.
• Construct a smaller real-world dataset with synchronized body-mounted (egocentric) and static (exocentric) camera recordings of diverse human actions.
• Train a conditional GAN (cGAN) to synthesize egocentric images from exocentric input images, conditioned on the scene and pose.
• Design a two-stream 3D CNN for cross-view retrieval using optical flow (2-channel) and RGB (3-channel) inputs to learn view-invariant features.
• Apply domain adaptation by fine-tuning the retrieval network trained on synthetic data using real egocentric data to improve generalization.
• Use SVMs on extracted features from the last fully connected layer to evaluate view-invariance and action recognition performance.

Experimental results

Research questions

• RQ1 Can a conditional GAN effectively synthesize plausible egocentric images from third-person video frames, even with limited visual overlap?
• RQ2 To what extent can synthetic egocentric data improve the performance of cross-view retrieval on real egocentric data?
• RQ3 Can a deep neural network learn view-invariant representations that generalize across egocentric and exocentric video domains?
• RQ4 How does domain adaptation from synthetic to real data affect retrieval accuracy in the egocentric-exocentric cross-view setting?
• RQ5 Does combining features from both egocentric and exocentric views improve action recognition performance compared to using either view alone?

Key findings

• The proposed cGAN-based synthesis network successfully hallucinates realistic egocentric views from exocentric inputs, preserving high-level semantic consistency.
• Retrieval performance on real egocentric data improved to 30.82% top-1 accuracy when using a retrieval network trained on synthetic data and fine-tuned on real data.
• The retrieval network trained on real RGB data achieved 42.58% top-1 accuracy, outperforming the synthetic-only baseline, indicating the value of real data for fine-tuning.
• Domain adaptation from synthetic to real data improved retrieval performance, with the best result (30.82%) achieved by a network trained on synthetic data and fine-tuned on real data.
• The view-invariant representation learned by the retrieval network preserved action recognition accuracy, with the combined view SVM achieving 30.82% on real data and outperforming individual view classifiers in some cases.
• The retrieval network trained on synthetic optical flow achieved 32.31% top-1 accuracy on real egocentric data, showing that synthetic data can effectively support motion-based retrieval.