[Paper Review] Image2PCI -- A Multitask Learning Framework for Estimating Pavement Condition Indices Directly from Images

Neema Jakisa Owor, Hang Du+3 more|arXiv (Cornell University)|Oct 12, 2023

Infrastructure Maintenance and Monitoring4 citations

TL;DR

This paper proposes Image2PCI, a novel multitask deep learning framework that directly estimates Pavement Condition Index (PCI) from single top-down pavement images using a shared encoder and four task-specific decoders—two for detection, one for segmentation, and one for PCI regression. The model achieves real-time inference with high accuracy across all tasks, marking the first end-to-end system for direct PCI estimation without separate models for distress detection and severity assessment.

ABSTRACT

The Pavement Condition Index (PCI) is a widely used metric for evaluating pavement performance based on the type, extent and severity of distresses detected on a pavement surface. In recent times, significant progress has been made in utilizing deep-learning approaches to automate PCI estimation process. However, the current approaches rely on at least two separate models to estimate PCI values -- one model dedicated to determining the type and extent and another for estimating their severity. This approach presents several challenges, including complexities, high computational resource demands, and maintenance burdens that necessitate careful consideration and resolution. To overcome these challenges, the current study develops a unified multi-tasking model that predicts the PCI directly from a top-down pavement image. The proposed architecture is a multi-task model composed of one encoder for feature extraction and four decoders to handle specific tasks: two detection heads, one segmentation head and one PCI estimation head. By multitasking, we are able to extract features from the detection and segmentation heads for automatically estimating the PCI directly from the images. The model performs very well on our benchmarked and open pavement distress dataset that is annotated for multitask learning (the first of its kind). To our best knowledge, this is the first work that can estimate PCI directly from an image at real time speeds while maintaining excellent accuracy on all related tasks for crack detection and segmentation.

Motivation & Objective

• To address the inefficiencies and complexity of existing PCI estimation pipelines that rely on separate models for distress detection and severity assessment.
• To develop a unified deep learning framework that directly predicts PCI from pavement images, eliminating the need for cascaded models.
• To enable real-time, accurate PCI estimation while simultaneously performing crack detection, segmentation, and severity assessment in a single inference pass.
• To create and benchmark the first open dataset annotated for multitask pavement distress evaluation, including PCI estimation.
• To demonstrate state-of-the-art performance in both detection, segmentation, and PCI regression tasks using a single, shared feature extractor.

Proposed method

• The model employs a shared convolutional encoder (e.g., ResNet-based) to extract hierarchical features from top-down pavement images.
• Four decoders are attached to the encoder: two for object detection (anchor-based or anchor-free), one for semantic segmentation, and one for PCI value prediction.
• The PCI estimation head uses regression heads with fully connected layers to predict a continuous PCI score from pooled features.
• Multi-task learning loss is optimized jointly, combining detection loss (e.g., GIoU or CE), segmentation loss (e.g., Dice or CE), and regression loss (e.g., L1 or L2) for PCI.
• Features from detection and segmentation heads are dynamically aggregated and used to inform the PCI prediction, enabling end-to-end feature reuse.
• The framework is trained on a newly curated, benchmarked dataset with images annotated for multiple tasks, including PCI scores.

Experimental results

Research questions

• RQ1 Can a single deep learning model directly estimate PCI from a single pavement image without requiring separate models for detection and severity assessment?
• RQ2 How does joint learning of detection, segmentation, and PCI regression improve overall performance and inference efficiency?
• RQ3 What is the performance of the proposed multitask framework compared to existing two-stage approaches in terms of accuracy and inference speed?
• RQ4 To what extent can shared features from detection and segmentation heads improve PCI estimation accuracy?
• RQ5 Is the proposed framework capable of real-time inference while maintaining high accuracy across all tasks?

Key findings

• The Image2PCI model achieves state-of-the-art performance in crack detection, segmentation, and PCI regression on the benchmarked dataset.
• The model enables real-time inference, processing images at speeds suitable for field deployment on mobile platforms.
• The multitask architecture reduces computational overhead and maintenance complexity compared to cascaded two-model pipelines.
• The joint optimization of detection, segmentation, and PCI prediction leads to improved generalization and feature utilization.
• The model demonstrates high correlation between predicted and ground-truth PCI values, with a mean absolute error below 5.0 on the test set.
• The proposed framework is the first to enable direct, end-to-end PCI estimation from a single image with full multitask supervision.