[Paper Review] Deep Learning for Rheumatoid Arthritis: Joint Detection and Damage Scoring in X-rays

Krzysztof Maziarz, Anna Krason+1 more|arXiv (Cornell University)|Apr 28, 2021

Radiomics and Machine Learning in Medical Imaging23 references4 citations

TL;DR

This paper presents a multi-task deep learning model that jointly detects joints and scores rheumatoid arthritis damage (narrowing and erosion) in hand and foot X-rays using a single convolutional neural network. By introducing local label smoothing to combine classification and regression signals, the model achieves a 5% relative error reduction and ranks 4th and 5th in the RA2 DREAM challenge for joint space narrowing and erosion scoring, respectively.

ABSTRACT

Recent advancements in computer vision promise to automate medical image analysis. Rheumatoid arthritis is an autoimmune disease that would profit from computer-based diagnosis, as there are no direct markers known, and doctors have to rely on manual inspection of X-ray images. In this work, we present a multi-task deep learning model that simultaneously learns to localize joints on X-ray images and diagnose two kinds of joint damage: narrowing and erosion. Additionally, we propose a modification of label smoothing, which combines classification and regression cues into a single loss and achieves 5% relative error reduction compared to standard loss functions. Our final model obtained 4th place in joint space narrowing and 5th place in joint erosion in the global RA2 DREAM challenge.

Motivation & Objective

• To automate rheumatoid arthritis diagnosis by enabling joint detection and damage scoring from X-ray images.
• To address the subjectivity and time intensity of manual radiographic assessment by replacing expert-dependent scoring with a data-driven approach.
• To improve model performance and generalization in low-data medical imaging by leveraging multi-task learning with joint localization and damage assessment.
• To reduce error in SvH score prediction through a novel label smoothing technique that integrates classification and regression signals.

Proposed method

• The model uses a U-Net-like encoder-decoder architecture with shared feature extraction for joint detection, erosion, and narrowing scoring.
• Joint center annotations are converted into segmentation masks using two radius parameters (r and R), enabling pixel-wise classification of joint regions.
• Coordinate channels (CoordConv) are added to the input to improve spatial awareness and localization accuracy.
• The final feature map is fed into three separate per-pixel classifiers for joint detection, erosion, and narrowing scoring.
• Local label smoothing is applied to the cross-entropy loss, redistributing 10% of the probability mass to adjacent classes to better align with regression objectives.
• During inference, SvH scores are computed as a weighted average using softmax probabilities from the model’s output.

Experimental results

Research questions

• RQ1 Can a single deep learning model effectively perform joint detection and dual damage scoring (narrowing and erosion) in hand and foot X-rays?
• RQ2 How does incorporating spatial localization cues through segmentation masks improve performance in low-data medical imaging?
• RQ3 To what extent can local label smoothing—blending classification and regression signals—reduce prediction error in SvH score estimation?
• RQ4 How does multi-task learning with shared feature extraction compare to separate models in terms of accuracy and efficiency?

Key findings

• The model achieved a mean RMSE of 0.4075 for joint space narrowing and 0.4607 for joint erosion on the test set, placing 4th and 5th, respectively, in the RA2 DREAM challenge.
• Local label smoothing reduced the relative error by approximately 5% compared to standard cross-entropy loss, demonstrating improved alignment with regression-based scoring.
• The optimal segmentation radius (r) was found to be around 32 pixels when R was fixed at 40, indicating sensitivity to spatial mask design.
• The model's performance was robust to data augmentation and achieved strong results with a simple, end-to-end multi-task architecture, outperforming more complex pipelines in some cases.
• The ablation study confirmed that local label smoothing and segmentation mask radius are critical hyperparameters, with significant impact on final model accuracy.