[Paper Review] Assessing Knee OA Severity with CNN attention-based end-to-end architectures

Marc Górriz, Joseph Antony+3 more|arXiv (Cornell University)|Aug 23, 2019

Osteoarthritis Treatment and Mechanisms33 references19 citations

TL;DR

This paper proposes an end-to-end CNN architecture with trainable attention modules that automatically localize and focus on relevant regions of interest (ROIs) in knee X-rays for osteoarthritis (OA) severity assessment. By integrating multi-scale attention mechanisms into pre-trained CNNs like VGG-16, the model achieves state-of-the-art performance with a Cohen’s kappa of 0.63, approaching human-level agreement on the OAI and MOST datasets.

ABSTRACT

This work proposes a novel end-to-end convolutional neural network (CNN) architecture to automatically quantify the severity of knee osteoarthritis (OA) using X-Ray images, which incorporates trainable attention modules acting as unsupervised fine-grained detectors of the region of interest (ROI). The proposed attention modules can be applied at different levels and scales across any CNN pipeline helping the network to learn relevant attention patterns over the most informative parts of the image at different resolutions. We test the proposed attention mechanism on existing state-of-the-art CNN architectures as our base models, achieving promising results on the benchmark knee OA datasets from the osteoarthritis initiative (OAI) and multicenter osteoarthritis study (MOST). All code from our experiments will be publicly available on the github repository: https://github.com/marc-gorriz/KneeOA-CNNAttention

Motivation & Objective

• To eliminate the need for separate knee joint localization steps in automated knee OA severity assessment.
• To improve classification performance by learning fine-grained, attention-driven feature patterns across multiple scales in CNNs.
• To develop a trainable, unsupervised attention mechanism that enhances model interpretability and robustness.
• To evaluate the method on benchmark datasets (OAI and MOST) and compare it to existing state-of-the-art approaches.
• To demonstrate that end-to-end training with multi-loss optimization improves attention mask quality and overall performance.

Proposed method

• Introduces a trainable attention module that operates on 2D feature maps at multiple layers and scales within a CNN pipeline.
• Applies the attention module as a learnable gate mechanism that generates spatial attention maps to highlight informative image regions.
• Uses a multi-loss training strategy to jointly optimize the main classification head and multiple attention branches with varying depths.
• Employs a weighted fusion of attention branches (e.g., w1 = 0.8 for deeper branch) to balance convergence speeds and improve feature abstraction.
• Adapts existing state-of-the-art CNNs (e.g., VGG-16) by inserting attention modules at intermediate layers without architectural overhaul.
• Trains the entire network end-to-end using cross-entropy and mean squared error losses for both classification and regression of KL grades.

Experimental results

Research questions

• RQ1 Can an end-to-end CNN with learnable attention mechanisms outperform two-stage localization-and-classification pipelines in knee OA severity grading?
• RQ2 How does multi-scale attention across different network depths improve feature learning and model performance?
• RQ3 To what extent can attention modules act as unsupervised ROI detectors without requiring annotated joint locations?
• RQ4 Does multi-loss training of multiple attention branches lead to better generalization and higher agreement with radiologists?
• RQ5 How does the proposed method compare to human-level inter-reader reliability in KL grading?

Key findings

• The proposed attention-based end-to-end model achieved a test accuracy of 64.3% on the OAI dataset, outperforming prior state-of-the-art methods.
• The model reached a Cohen’s kappa of 0.63 with radiologist annotations, indicating substantial agreement and approaching human-level reliability.
• The best performance was achieved using a multi-loss strategy with reduced contribution from deeper attention branches (w1 = 0.8), improving mask precision.
• The method eliminated the need for separate knee joint localization, reducing training complexity and dependency on manual annotations.
• Attention modules effectively localized relevant anatomical structures (e.g., joint space, osteophytes) without supervision, demonstrating robustness to noise and variability.
• The approach generalizes across different base architectures and shows consistent improvement over baseline models like Antony et al. (2017) and VGG-16 finetuned without attention.