[Paper Review] CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation

Haisong Liu, Tao Lu+4 more|arXiv (Cornell University)|Nov 20, 2021

Advanced Vision and Imaging52 references4 citations

TL;DR

CamLiFlow proposes an end-to-end, bidirectional camera-LiDAR fusion framework for joint optical and scene flow estimation, using modality-specific 2D and 3D branches with learnable, symmetric fusion modules (Bi-CLFM) and inverse depth scaling to balance sparse point clouds. It achieves state-of-the-art performance on KITTI and FlyingThings3D with 1/7 and 1/6 parameters respectively, ranking first on the KITTI Scene Flow benchmark.

ABSTRACT

In this paper, we study the problem of jointly estimating the optical flow and scene flow from synchronized 2D and 3D data. Previous methods either employ a complex pipeline that splits the joint task into independent stages, or fuse 2D and 3D information in an "early-fusion" or "late-fusion" manner. Such one-size-fits-all approaches suffer from a dilemma of failing to fully utilize the characteristic of each modality or to maximize the inter-modality complementarity. To address the problem, we propose a novel end-to-end framework, called CamLiFlow. It consists of 2D and 3D branches with multiple bidirectional connections between them in specific layers. Different from previous work, we apply a point-based 3D branch to better extract the geometric features and design a symmetric learnable operator to fuse dense image features and sparse point features. Experiments show that CamLiFlow achieves better performance with fewer parameters. Our method ranks 1st on the KITTI Scene Flow benchmark, outperforming the previous art with 1/7 parameters. Code is available at https://github.com/MCG-NJU/CamLiFlow.

Motivation & Objective

• To address the limitations of prior methods that use early/late fusion or modular pipelines for joint optical and scene flow estimation.
• To enable effective, bidirectional feature fusion between dense 2D image features and sparse 3D LiDAR point features despite their structural and density differences.
• To improve performance by leveraging modality-specific architectures and learnable fusion mechanisms that maximize inter-modality complementarity.
• To reduce model complexity while maintaining or improving accuracy through symmetric, multi-stage fusion and inverse depth scaling for point cloud balancing.

Proposed method

• CamLiFlow employs separate 2D and 3D branches based on PWC-Net and point-based networks to preserve modality-specific feature extraction capabilities.
• It introduces a bidirectional camera-LiDAR fusion module (Bi-CLFM) that enables learnable, symmetric feature fusion in both directions using interpolation and sampling with learnable weights.
• The Bi-CLFM incorporates 2D image similarity measurements into the interpolation process to improve robustness in scenes with overlapping objects.
• It applies inverse depth scaling (IDS) to non-linearly rescale LiDAR points based on inverse depth, balancing point density across distances and improving feature learning.
• Fusion is performed at multiple stages—feature extraction, cost volume, and flow decoding—enabling iterative refinement and better cross-modal information exchange.
• The framework is trained end-to-end with a symmetric learnable operator that ensures consistent feature interaction between image and point branches.

Experimental results

Research questions

• RQ1 Can a multi-stage, bidirectional fusion strategy outperform early or late fusion in joint optical and scene flow estimation?
• RQ2 How can dense 2D image features and sparse 3D LiDAR point features be effectively fused when they lack one-to-one correspondence and vary in density?
• RQ3 Does incorporating 2D image similarity into the fusion process improve robustness in complex scenes with occlusions?
• RQ4 Can inverse depth scaling improve feature learning on sparse LiDAR point clouds by balancing their spatial distribution?
• RQ5 Can a lightweight, end-to-end model achieve state-of-the-art performance with significantly fewer parameters than prior methods?

Key findings

• CamLiFlow achieves a 4.43% end-to-end error on the KITTI Scene Flow benchmark, outperforming the previous state-of-the-art method with only 1/7 parameters.
• On FlyingThings3D, CamLiFlow reduces end-point error by 48.4% compared to RAFT-3D while using only 1/6 of the parameters.
• The introduction of fusion-aware interpolation with 2D similarity measurements reduces EPE 2D from 2.30 to 2.18 and improves ACC 1px from 83.3% to 84.5%.
• Inverse depth scaling (IDS) improves performance by 2.7% in EPE 3D and 2.7 percentage points in ACC .05 on the LiDAR-only variant, demonstrating its effectiveness for point cloud balancing.
• Fusing features at all three stages (feature extraction, cost volume, flow decoding) yields the best performance, outperforming early/late fusion and single-stage fusion.
• The model achieves 118ms inference time on 960x540 images using a Tesla V100 GPU, with the point branch being the most time-consuming component.