[Paper Review] Quality Assessment of Image Matchers for DSM Generation - A Comparative Study Based on UAV Images

Rongjun Qin, Armin Gruen+1 more|arXiv (Cornell University)|Jan 1, 2014

3D Surveying and Cultural Heritage5 citations

TL;DR

This study evaluates five image matching software packages—APS, DLR-SGM, MICMAC, Pix4D, and SURE—for generating high-resolution Digital Surface Models (DSMs) from 5 cm GSD UAV imagery. Using a reference 3D model derived from mobile LiDAR and manual stereo measurements, DSMs were co-registered via least-squares 3D matching (LS3D), revealing SURE as the most accurate with the lowest RMSE (0.24 m) and standard deviation (0.24 m) in buildings-only scenarios.

ABSTRACT

Recently developed automatic dense image matching algorithms are now being implemented for DSM/DTM production, with their pixel-level surface generation capability offering the prospect of partially alleviating the need for manual and semi-automatic stereoscopic measurements. In this paper, five commercial/public software packages for 3D surface generation are evaluated, using 5cm GSD imagery recorded from a UAV. Generated surface models are assessed against point clouds generated from mobile LiDAR and manual stereoscopic measurements. The software packages considered are APS, MICMAC, SURE, Pix4UAV and an SGM implementation from DLR.

Motivation & Objective

• To assess the performance of five commercial and open-source dense image matching (DIM) software packages in generating accurate DSMs from high-resolution UAV imagery.
• To evaluate matcher accuracy under real-world urban conditions, including complex textures, vegetation, and man-made structures.
• To quantify errors and blunders in DSMs using a high-accuracy reference 3D model derived from mobile LiDAR and manual stereo measurements.
• To provide a benchmark for matcher selection in high-resolution UAV-based 3D urban modeling.

Proposed method

• Five DIM software packages—APS, DLR-SGM, MICMAC, Pix4D, and SURE—were applied to 5 cm GSD vertical UAV images of a 354 m × 185 m urban test area.
• A reference 3D surface model was constructed using mobile LiDAR point clouds and manual stereo measurements from UAV images.
• DSMs generated by each matcher were co-registered to the reference model using the LS3D software, which minimizes the sum of squared Euclidean distances between matched surface points.
• Accuracy was evaluated using RMSE, standard deviation (STD), mean error, and blunder detection (errors >3×STD), across three scenarios: full DSM, DSM without trees, and buildings-only.
• Error distributions were analyzed via histograms and visualized on orthophotos to identify spatial patterns of blunders and gaps.
• Blunders were identified as points with residuals exceeding 3 times the standard deviation of the co-registration process.

Experimental results

Research questions

• RQ1 Which image matcher produces the most accurate DSM when applied to 5 cm GSD UAV imagery in a complex urban environment?
• RQ2 How do matcher performance metrics (RMSE, STD, blunder rate) vary across different surface types—buildings, roads, terrain, and vegetation?
• RQ3 What is the impact of masking vegetation on matcher accuracy, and how does performance differ in buildings-only scenarios?
• RQ4 How do error distributions and blunder patterns vary between matchers, particularly in textured and homogeneous regions?
• RQ5 To what extent do high-resolution UAV images (5 cm GSD) challenge current DIM algorithms, and what are the main failure modes?

Key findings

• SURE achieved the lowest RMSE of 0.24 m and standard deviation of 0.24 m in the buildings-only scenario, outperforming all other matchers.
• The overall RMSE for all matchers ranged from 0.24 m (SURE) to 0.51 m (APS), with SURE also showing the smallest standard deviation across all test cases.
• Blunder rates varied significantly: Pix4D had the highest blunder percentage (2.77%) in the full DSM, while DLR-SGM had the lowest (2.19%), though absolute numbers remained high (e.g., 97,903 blunders for DLR-SGM).
• All matchers performed best on buildings-only surfaces, with RMSE decreasing from ~0.5 m (full DSM) to ~0.24 m (buildings only), indicating that vegetation and complex textures degrade performance.
• Large blunders (>0.8 m) were predominantly located in vegetated areas and on building edges, especially around homogeneous textures like flat roofs and swimming pools.
• Despite high-resolution input (5 cm GSD), the best-performing matcher (SURE) still achieved an accuracy level of 3–5 pixels (15–25 cm) after blunder removal, indicating persistent challenges in dense matching.