[Paper Review] Clinical-Prior Guided Multi-Modal Learning with Latent Attention Pooling for Gait-Based Scoliosis Screening

Dong Chen, Zizhuang Wei+7 more|arXiv (Cornell University)|Feb 6, 2026

Scoliosis diagnosis and treatment0 citations

TL;DR

Introduces ScoliGait, a non-overlapping, radiographically labeled gait video benchmark for AIS screening, and a clinical-prior guided multi-modal model with latent attention pooling that fuses knowledge map, video, and text for interpretable, state-of-the-art performance.

ABSTRACT

Adolescent Idiopathic Scoliosis (AIS) is a prevalent spinal deformity whose progression can be mitigated through early detection. Conventional screening methods are often subjective, difficult to scale, and reliant on specialized clinical expertise. Video-based gait analysis offers a promising alternative, but current datasets and methods frequently suffer from data leakage, where performance is inflated by repeated clips from the same individual, or employ oversimplified models that lack clinical interpretability. To address these limitations, we introduce ScoliGait, a new benchmark dataset comprising 1,572 gait video clips for training and 300 fully independent clips for testing. Each clip is annotated with radiographic Cobb angles and descriptive text based on clinical kinematic priors. We propose a multi-modal framework that integrates a clinical-prior-guided kinematic knowledge map for interpretable feature representation, alongside a latent attention pooling mechanism to fuse video, text, and knowledge map modalities. Our method establishes a new state-of-the-art, demonstrating a significant performance gap on a realistic, non-repeating subject benchmark. Our approach establishes a new state of the art, showing a significant performance gain on a realistic, subject-independent benchmark. This work provides a robust, interpretable, and clinically grounded foundation for scalable, non-invasive AIS assessment.

Motivation & Objective

• Address data leakage and subject-independence in gait-based AIS screening datasets.
• Provide a clinically grounded, interpretable representation of gait through a kinematic knowledge map.
• Develop a robust multi-modal fusion method that combines video, knowledge map, and text with latent attention pooling.

Proposed method

• Propose ScoliGait dataset with 1,572 training clips and 300 independent test clips, each annotated with radiographic Cobb angles and clinical text prompts.
• Construct a kinematic knowledge map with 238 features across motion space, self-skeleton space, and signal cross-correlation.
• Use three modality-specific encoders (knowledge map, video, text) and a latent attention pooling mechanism to fuse modalities.
• Align positional embeddings across modalities to improve fusion performance.
• Provide interpretable explanations by mapping attention scores back to the clinical knowledge map.
• Adopt Sentence-Transformers for text encoding and Vision Transformer backbones for video and knowledge map modalities.

Figure 1: ScoliGait system for multi-modal gait analysis from mobile video. Left: temporal alignment of the knowledge map and video. Right: generation of video, knowledge map, and text modalities via pose estimation, showing kinematic alignment and knowledge-guided synthesis.

Experimental results

Research questions

• RQ1 Can a clinically grounded, multi-modal framework improve AIS screening accuracy on a subject-independent gait dataset?
• RQ2 Does integrating a structured kinematic knowledge map with video and text enhance interpretability and diagnostic performance for scoliosis screening?
• RQ3 What is the impact of latent attention pooling and cross-modal alignment on fusion quality and clinical relevance?

Key findings

• The knowledge map alone outperforms video alone in binary AIS screening by 1.7% accuracy and 3.2% F1-score.
• Multimodal fusion of knowledge map, video, and text with latent attention pooling achieves the best performance: accuracy 70.0% and F1-score 61.9%.
• ScoliGait provides a non-overlapping, radiographically labeled benchmark with 1,572 training clips and 300 independent test clips from unique individuals.
• Explainability is enhanced by mapping attention to a clinically meaningful knowledge map, enabling explicit interpretation of gait features over time.
• Ablation shows latent attention pooling outperforms simple concatenation and that cross-modal embedding alignment improves results.

Figure 2: Proposed three-modal fusion architecture for AIS screening. Inputs from Knowledge Map, Vision, and Text modalities are integrated via a Latent Attention Pooling mechanism (bottom). Remapped attention scores from the Knowledge Map (top) are filtered for salient features to enable clinical i