[Paper Review] Weakly-Supervised Localization and Classification of Proximal Femur Fractures

Amelia Jiménez-Sánchez, Anees Kazi+6 more|arXiv (Cornell University)|Sep 27, 2018

COVID-19 diagnosis using AI15 references3 citations

TL;DR

This paper proposes a weakly-supervised deep learning framework for joint localization and classification of proximal femur fractures in X-ray images using only image-level labels. By integrating spatial transformers, self-transfer learning, and global pooling-based localization, the method achieves improved classification accuracy—up to 7% higher than baseline—without requiring additional bounding box annotations, demonstrating that weakly-supervised localization enhances both performance and interpretability in hierarchical AO fracture classification.

ABSTRACT

In this paper, we target the problem of fracture classification from clinical X-Ray images towards an automated Computer Aided Diagnosis (CAD) system. Although primarily dealing with an image classification problem, we argue that localizing the fracture in the image is crucial to make good class predictions. Therefore, we propose and thoroughly analyze several schemes for simultaneous fracture localization and classification. We show that using an auxiliary localization task, in general, improves the classification performance. Moreover, it is possible to avoid the need for additional localization annotations thanks to recent advancements in weakly-supervised deep learning approaches. Among such approaches, we investigate and adapt Spatial Transformers (ST), Self-Transfer Learning (STL), and localization from global pooling layers. We provide a detailed quantitative and qualitative validation on a dataset of 1347 femur fractures images and report high accuracy with regard to inter-expert correlation values reported in the literature. Our investigations show that i) lesion localization improves the classification outcome, ii) weakly-supervised methods improve baseline classification without any additional cost, iii) STL guides feature activations and boost performance. We plan to make both the dataset and code available.

Motivation & Objective

• To improve the accuracy of proximal femur fracture classification according to the AO system using deep learning.
• To investigate whether localizing the fracture region enhances classification performance compared to full-image classification.
• To reduce annotation burden by leveraging weakly-supervised methods that infer localization from image-level labels only.
• To evaluate and compare multiple weakly-supervised localization techniques in the context of medical image classification.
• To enhance model interpretability through attention maps and localization heatmaps without additional supervision.

Proposed method

• Utilizes a multi-task learning framework where classification and localization are jointly optimized using deep neural networks.
• Employs Spatial Transformers (ST) to learn spatial transformations that focus on the fracture region from the input X-ray.
• Adapts Self-Transfer Learning (STL) to enable end-to-end training of localization and classification in a parallel, differentiable manner.
• Explores localization via global pooling layers as a lightweight alternative to dedicated localization heads.
• Implements a baseline model (LBM) using only the full image for classification, serving as a performance reference.
• Uses a weakly-supervised training scheme where gradients from the classification loss backpropagate to learn localization implicitly, avoiding the need for bounding box annotations.

Experimental results

Research questions

• RQ1 Does incorporating weakly-supervised localization improve the performance of AO fracture classification in proximal femur X-rays?
• RQ2 How does the performance of weakly-supervised localization methods compare to fully supervised localization in terms of classification accuracy and localization quality?
• RQ3 To what extent does attention-based localization enhance model interpretability and decision transparency?
• RQ4 Can self-attention mechanisms like Self-Transfer Learning (STL) guide feature learning and improve classification beyond standard weakly-supervised baselines?
• RQ5 What is the impact of localization on fine-grained classification (6-class AO system), especially given low inter-expert agreement?

Key findings

• Localization significantly improves classification performance, with an average 9% accuracy gain when using ground-truth bounding boxes compared to full-image classification.
• Weakly-supervised methods such as Self-Transfer Learning (STL) boost classification performance by 7% over the baseline model (LBM) without any additional annotations.
• The best-performing model achieved a mean average precision (mAP) of 0.47 for localization, though the IoU-based metric may be overly sensitive to scale errors.
• Models using weakly-supervised localization produce interpretable attention maps that highlight fracture regions, improving clinical trust and model transparency.
• STL outperforms other weakly-supervised methods in both localization and classification, likely due to its parallel, joint optimization of localization and classification.
• The Sup. Loc. model (with ground-truth boxes) achieved the highest performance, confirming that localization is crucial for fine-grained AO classification, especially in the 6-class scenario where performance remains below inter-observer variability (66–71%).