[Paper Review] RecRecNet: Rectangling Rectified Wide-Angle Images by Thin-Plate Spline Model and DoF-based Curriculum Learning

Kang Liao, Lang Nie+3 more|arXiv (Cornell University)|Jan 4, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes RecRecNet, a novel unsupervised deep learning framework that rectifies wide-angle images by jointly preserving undistorted content and straight boundaries using a thin-plate spline (TPS) module for non-rigid deformation and a degree-of-freedom (DoF)-based curriculum learning strategy. The method achieves superior performance in both quantitative metrics and visual quality by progressively learning deformation rules from 4-DoF similarity to 8-DoF homography transformations.

ABSTRACT

The wide-angle lens shows appealing applications in VR technologies, but it introduces severe radial distortion into its captured image. To recover the realistic scene, previous works devote to rectifying the content of the wide-angle image. However, such a rectification solution inevitably distorts the image boundary, which changes related geometric distributions and misleads the current vision perception models. In this work, we explore constructing a win-win representation on both content and boundary by contributing a new learning model, i.e., Rectangling Rectification Network (RecRecNet). In particular, we propose a thin-plate spline (TPS) module to formulate the non-linear and non-rigid transformation for rectangling images. By learning the control points on the rectified image, our model can flexibly warp the source structure to the target domain and achieves an end-to-end unsupervised deformation. To relieve the complexity of structure approximation, we then inspire our RecRecNet to learn the gradual deformation rules with a DoF (Degree of Freedom)-based curriculum learning. By increasing the DoF in each curriculum stage, namely, from similarity transformation (4-DoF) to homography transformation (8-DoF), the network is capable of investigating more detailed deformations, offering fast convergence on the final rectangling task. Experiments show the superiority of our solution over the compared methods on both quantitative and qualitative evaluations. The code and dataset are available at https://github.com/KangLiao929/RecRecNet.

Motivation & Objective

• Address the critical issue that conventional rectification methods distort image boundaries, degrading performance of downstream vision perception models.
• Propose a win-win representation that preserves both geometrically accurate content and regular, straight boundaries in rectified wide-angle images.
• Develop a flexible, end-to-end unsupervised learning framework capable of modeling complex non-linear and non-rigid deformations.
• Improve training stability and convergence speed by introducing a DoF-based curriculum learning strategy that progresses from simple to complex transformations.
• Investigate the root cause of performance degradation in vision models due to deformed image boundaries and demonstrate its widespread impact across vision tasks.

Proposed method

• Employ a thin-plate spline (TPS) motion module that learns control points on the rectified image to model non-linear, non-rigid warping from distorted to rectified space.
• Formulate the deformation as a differentiable transformation using TPS, enabling end-to-end training without ground-truth supervision.
• Implement a DoF-based curriculum learning schedule that starts from 4-DoF similarity transformations and gradually increases to 8-DoF homography transformations.
• Structure the curriculum to progress from straight to curved boundaries, enabling the network to learn hierarchical geometric rules and avoid poor local minima.
• Use a self-supervised loss based on image reconstruction and structural consistency to optimize the TPS deformation without requiring paired real-world data.
• Design the training pipeline to initialize with low-complexity transformations and progressively refine the deformation capability, improving convergence and generalization.

Experimental results

Research questions

• RQ1 Why do deformed boundaries in rectified wide-angle images cause significant performance degradation in modern vision perception models like Mask R-CNN?
• RQ2 Can a unified representation be learned that simultaneously rectifies image content and preserves straight, regular image boundaries?
• RQ3 How does progressive, DoF-based curriculum learning improve the training of non-rigid deformation networks for wide-angle image rectification?
• RQ4 To what extent does the TPS-based deformation model outperform conventional rectification methods in preserving geometric fidelity and semantic consistency?
• RQ5 Does the proposed method generalize across diverse real-world wide-angle lenses and synthesized datasets without domain-specific fine-tuning?

Key findings

• The TPS module with a $9\times9$ control point grid achieves the best performance, yielding a PSNR of 18.68, SSIM of 0.5450, FID of 19.01, and LPIPS of 0.1136 on the benchmark dataset.
• Curriculum learning with a 4-DoF to 8-DoF progression (Vanilla + C3) enables faster convergence and better generalization, reducing training epochs by ~40 compared to vanilla training.
• The ablation study confirms that increasing control point density improves boundary localization and structure recovery, with $9\times9$ being optimal.
• RecRecNet generalizes well to real-world wide-angle images from diverse lenses (e.g., Rokinon 8mm, GoPro), outperforming state-of-the-art methods on cross-domain evaluation.
• The study reveals that deformed boundaries introduce artificial edges and features in feature maps, which propagate through convolutional layers and cause semantic misinterpretation in models like Mask R-CNN.
• The edge effect from deformed boundaries is not limited to rectification but also affects other vision tasks such as image stitching, warping, and roll shutter correction, highlighting its broader impact.