[Paper Review] Multi-Scale RAFT: Combining Hierarchical Concepts for Learning-based Optical FLow Estimation

Azin Jahedi, Lukas Mehl+2 more|arXiv (Cornell University)|Jul 25, 2022

Retinal Imaging and Analysis4 citations

TL;DR

This paper proposes Multi-Scale RAFT (MS-RAFT), a novel optical flow network that enhances the single-scale RAFT framework by integrating coarse-to-fine estimation, U-Net-style multi-scale features, a correlation pyramid, and a multi-scale multi-iteration loss. The method achieves state-of-the-art performance on MPI Sintel and KITTI benchmarks, particularly in non-occluded regions, outperforming RAFT by up to 14.6% in accuracy on Sintel Final with improved robustness and generalization.

ABSTRACT

Many classical and learning-based optical flow methods rely on hierarchical concepts to improve both accuracy and robustness. However, one of the currently most successful approaches -- RAFT -- hardly exploits such concepts. In this work, we show that multi-scale ideas are still valuable. More precisely, using RAFT as a baseline, we propose a novel multi-scale neural network that combines several hierarchical concepts within a single estimation framework. These concepts include (i) a partially shared coarse-to-fine architecture, (ii) multi-scale features, (iii) a hierarchical cost volume and (iv) a multi-scale multi-iteration loss. Experiments on MPI Sintel and KITTI clearly demonstrate the benefits of our approach. They show not only substantial improvements compared to RAFT, but also state-of-the-art results -- in particular in non-occluded regions. Code will be available at https://github.com/cv-stuttgart/MS_RAFT.

Motivation & Objective

• To improve the performance of RAFT, a highly successful single-scale optical flow network, by incorporating hierarchical concepts from prior methods.
• To investigate whether multi-scale design principles—previously effective in methods like PWC-Net and SpyNet—remain beneficial when applied to the RAFT architecture.
• To enhance flow estimation accuracy and robustness, especially in non-occluded regions, through integrated multi-scale feature learning and loss supervision.
• To develop a more robust training objective by introducing a sample-wise robustified multi-scale multi-iteration loss.

Proposed method

• Introduces a shared, partially shared coarse-to-fine architecture where flow estimates are iteratively refined from coarse to fine scales, using upsampling with learned convex masks.
• Employs U-Net-style skip connections to integrate multi-scale features from the encoder, enabling semantic enrichment across scales during matching.
• Retains RAFT’s correlation pyramid as a hierarchical cost volume to inject non-local cost information at each scale, improving matching accuracy.
• Proposes a multi-scale multi-iteration loss that applies supervision across all scales and iterations, with sample-wise robustification to improve training stability and generalization.
• Uses a shared recurrent update unit across scales, enabling parameter efficiency while maintaining high performance.

Experimental results

Research questions

• RQ1 Can multi-scale design principles—such as coarse-to-fine estimation and multi-scale features—still improve performance when applied to the single-scale RAFT framework?
• RQ2 How does combining hierarchical concepts (coarse-to-fine, multi-scale features, correlation pyramid, multi-scale loss) impact flow estimation accuracy and robustness?
• RQ3 Does a multi-scale multi-iteration loss with sample-wise robustification lead to better generalization and reduced overfitting compared to single-scale supervision?
• RQ4 To what extent do the proposed components improve performance in challenging regions, such as non-occluded areas?

Key findings

• MS-RAFT achieves a 14.6% relative improvement in end-point error (EPE) on the Sintel Final benchmark compared to RAFT, with an EPE of 1.37.
• On the Sintel Clean benchmark, MS-RAFT reduces EPE to 1.37, a 14.6% improvement over RAFT’s 1.61 EPE.
• In non-occluded regions, MS-RAFT achieves state-of-the-art performance, demonstrating superior accuracy and robustness.
• The ablation study confirms that the combination of coarse-to-fine architecture, multi-scale features, and multi-scale loss is essential—no single component yields significant gains alone.
• The multi-scale multi-iteration loss with sample-wise robustification reduces overfitting and improves generalization, outperforming a single-scale loss by 43.6% in EPE on Sintel Final.
• The best-performing configuration uses 3 scales with 2 lookup levels in the correlation pyramid, balancing accuracy and computational cost.