[Paper Review] Deep ChArUco: Dark ChArUco Marker Pose Estimation

Danying Hu, Daniel DeTone+3 more|arXiv (Cornell University)|Dec 8, 2018

Robotics and Sensor-Based Localization17 references4 citations

TL;DR

This paper proposes Deep ChArUco, a real-time, deep learning-based system for robust 6DoF pose estimation of ChArUco markers under extreme lighting and motion conditions where traditional OpenCV methods fail. It combines a two-headed CNN (ChArUcoNet) for joint ID classification and 2D corner detection with a subpixel refinement network (RefineNet), trained on synthetic and auto-labeled data with extreme augmentation, achieving superior accuracy and robustness in low-light and high-blur scenarios.

ABSTRACT

ChArUco boards are used for camera calibration, monocular pose estimation, and pose verification in both robotics and augmented reality. Such fiducials are detectable via traditional computer vision methods (as found in OpenCV) in well-lit environments, but classical methods fail when the lighting is poor or when the image undergoes extreme motion blur. We present Deep ChArUco, a real-time pose estimation system which combines two custom deep networks, ChArUcoNet and RefineNet, with the Perspective-n-Point (PnP) algorithm to estimate the marker's 6DoF pose. ChArUcoNet is a two-headed marker-specific convolutional neural network (CNN) which jointly outputs ID-specific classifiers and 2D point locations. The 2D point locations are further refined into subpixel coordinates using RefineNet. Our networks are trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and extreme data augmentation. We evaluate Deep ChArUco in challenging low-light, high-motion, high-blur scenarios and demonstrate that our approach is superior to a traditional OpenCV-based method for ChArUco marker detection and pose estimation.

Motivation & Objective

• Address the failure of traditional OpenCV-based ChArUco detection in low-light and high-motion blur conditions.
• Develop a real-time, robust marker detection system for augmented reality and robotics applications under challenging visual conditions.
• Improve pose estimation accuracy by combining deep learning with classical PnP algorithms for 6DoF pose recovery.
• Design a training pipeline that leverages synthetic data and auto-labeled real videos to overcome data scarcity for rare visual conditions.
• Enable reliable detection of ChArUco markers even when classical methods fail due to shadows, low illumination, or motion blur.

Proposed method

• Propose ChArUcoNet, a two-headed fully convolutional neural network that jointly predicts marker ID and 2D corner locations for each detected ChArUco pattern.
• Introduce RefineNet, a subpixel refinement network that enhances corner localization accuracy by refining coarse detections from ChArUcoNet using a learned regression on a local 8×8 region.
• Train the networks using a hybrid data pipeline combining auto-labeled real video sequences of the target marker with synthetic subpixel-accurate corner data.
• Apply extreme data augmentation, including random lighting variations, motion blur, and shadow simulation, to improve robustness to adverse conditions.
• Use the Perspective-n-Point (PnP) algorithm to compute the 6DoF pose from the refined 2D-3D correspondences obtained by the deep networks.
• Enable real-time inference by leveraging the fully convolutional nature of ChArUcoNet, allowing processing at various resolutions with optional high-resolution refinement via RefineNet.

Experimental results

Research questions

• RQ1 Can deep learning improve ChArUco marker detection and pose estimation robustness in low-light and high-motion blur conditions where OpenCV fails?
• RQ2 How effective is a two-stage deep learning pipeline (detection + subpixel refinement) compared to classical corner refinement in extreme visual conditions?
• RQ3 To what extent can synthetic data and auto-labeled real data reduce the need for manual annotation while maintaining high performance?
• RQ4 Does the use of pattern-specific networks with extreme data augmentation significantly outperform general-purpose detection methods in challenging scenarios?
• RQ5 Can the proposed system maintain real-time performance while achieving subpixel accuracy under adverse lighting and motion conditions?

Key findings

• Deep ChArUco achieves 100% detection accuracy in 26 test sequences under 3 lux or higher illumination, while OpenCV fails entirely at 1 lux and below.
• In shadowed conditions (e.g., 100 lux with cast shadows), Deep ChArUco maintains 100% accuracy with a mean reprojection error of 0.102 pixels, compared to OpenCV’s 30.1% accuracy and 0.116 error.
• Under motion blur, Deep ChArUco achieves 80.3% accuracy with a mean reprojection error of 1.347 pixels, significantly outperforming OpenCV’s 31.1% accuracy and 0.160 error.
• RefineNet reduces reprojection error in all but motion-blur cases, with the largest improvement observed in low-light conditions (e.g., 0.858 to 0.427 pixels at 0.3 lux).
• The system runs in real time, processing 320×240 images at over 30 FPS on a GTX 1080 GPU, with inference time of 98.6 ms when using RefineNet.
• The combination of auto-labeled videos and synthetic subpixel data enables high-performance training without extensive manual annotation, validating the data pipeline’s effectiveness.