[Paper Review] Future Segmentation Using 3D Structure

Suhani Vora, Reza Mahjourian+2 more|arXiv (Cornell University)|Nov 28, 2018

Robotics and Sensor-Based Localization22 references4 citations

TL;DR

This paper proposes a novel approach to future semantic segmentation by leveraging 3D scene structure through learned depth estimation and ego-motion prediction. By transforming past semantic masks via predicted 3D rigid transformations (SE3), the method achieves state-of-the-art performance, improving IoU by up to 12.5% over baselines and enabling accurate predictions up to 0.9 seconds ahead without future RGB or motion data.

ABSTRACT

Predicting the future to anticipate the outcome of events and actions is a critical attribute of autonomous agents; particularly for agents which must rely heavily on real time visual data for decision making. Working towards this capability, we address the task of predicting future frame segmentation from a stream of monocular video by leveraging the 3D structure of the scene. Our framework is based on learnable sub-modules capable of predicting pixel-wise scene semantic labels, depth, and camera ego-motion of adjacent frames. We further propose a recurrent neural network based model capable of predicting future ego-motion trajectory as a function of a series of past ego-motion steps. Ultimately, we observe that leveraging 3D structure in the model facilitates successful prediction, achieving state of the art accuracy in future semantic segmentation.

Motivation & Objective

• To improve future semantic segmentation by incorporating 3D scene geometry instead of relying solely on RGB or 2D spatial features.
• To enable accurate long-horizon future frame prediction by learning 3D transformations from monocular video without external sensors.
• To reduce blurring and uncertainty in future predictions by operating in the semantic and geometric space rather than raw pixel space.
• To develop a self-supervised framework that jointly learns depth, ego-motion, and future trajectory from video alone.
• To demonstrate that 3D structure and motion modeling significantly enhance segmentation accuracy for future frames.

Proposed method

• The method uses a joint model to predict depth and ego-motion from monocular RGB video, enabling 3D reconstruction of the scene from past frames.
• Semantic segmentation masks from past frames are projected into 3D space using predicted depth, forming a 3D point cloud.
• The 3D point cloud is transformed via predicted future ego-motion (SE3 transformation), simulating the agent's future viewpoint.
• The transformed 3D point cloud is then projected back into 2D image space to generate the future segmentation mask.
• A recurrent neural network (RNN) is trained to predict future ego-motion trajectories from past ego-motion sequences, improving long-term accuracy.
• An inpainting module is applied after each transformation step to mitigate error accumulation from missing or invalid depth values.

Experimental results

Research questions

• RQ1 Can 3D geometric structure improve the accuracy of future semantic segmentation beyond 2D pixel-level prediction?
• RQ2 How does learning future ego-motion trajectories from past motion sequences affect long-term segmentation performance?
• RQ3 To what extent does incorporating 3D depth and motion reduce blurring and uncertainty in future frame predictions?
• RQ4 Can a self-supervised model learn depth and ego-motion from monocular video alone to enable accurate future segmentation?
• RQ5 Does using 3D rigid transformations (SE3) provide a more structured and reliable path for future prediction than direct autoregressive or GAN-based methods?

Key findings

• The method achieves state-of-the-art performance, with a mean IoU of 0.6147 on the 16→19 prediction task when combining motion transformation, inpainting, and learned future ego-motion.
• Incorporating learned future ego-motion improves IoU by 1.5% over the motion transformation + inpainting baseline on the 16→19 task.
• The model achieves a 36.8% IoU on the 4→19 task when using estimated ego-motion, demonstrating that accurate motion prediction is critical for long-term performance.
• Inpainting reduces error accumulation and improves results, especially on longer time horizons (e.g., 16→19), where missing values are more prevalent.
• Error analysis shows that the main sources of error are under-estimated ego-motion and occlusions, indicating room for improvement in motion prediction and geometry modeling.
• The method successfully predicts semantic segmentation up to 0.9 seconds into the future, demonstrating feasibility for real-time autonomous decision-making.