[Paper Review] METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens

Zhanyu Wang, Lingqiao Liu+2 more|arXiv (Cornell University)|Apr 5, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

METransformer proposes a multi-expert attention mechanism in a transformer-based model for radiology report generation, introducing learnable expert tokens in both encoder and decoder to enable complementary, diverse attention to image regions. By using orthogonal loss to encourage distinct focus and a metrics-based voting strategy for final report selection, it achieves state-of-the-art performance on IU-Xray and MIMIC-CXR with minimal parameter increase, outperforming ensemble methods in efficiency and accuracy.

ABSTRACT

In clinical scenarios, multi-specialist consultation could significantly benefit the diagnosis, especially for intricate cases. This inspires us to explore a "multi-expert joint diagnosis" mechanism to upgrade the existing "single expert" framework commonly seen in the current literature. To this end, we propose METransformer, a method to realize this idea with a transformer-based backbone. The key design of our method is the introduction of multiple learnable "expert" tokens into both the transformer encoder and decoder. In the encoder, each expert token interacts with both vision tokens and other expert tokens to learn to attend different image regions for image representation. These expert tokens are encouraged to capture complementary information by an orthogonal loss that minimizes their overlap. In the decoder, each attended expert token guides the cross-attention between input words and visual tokens, thus influencing the generated report. A metrics-based expert voting strategy is further developed to generate the final report. By the multi-experts concept, our model enjoys the merits of an ensemble-based approach but through a manner that is computationally more efficient and supports more sophisticated interactions among experts. Experimental results demonstrate the promising performance of our proposed model on two widely used benchmarks. Last but not least, the framework-level innovation makes our work ready to incorporate advances on existing "single-expert" models to further improve its performance.

Motivation & Objective

• To address the challenge of fine-grained attention in radiology report generation, where anomalies are small and location-agnostic.
• To simulate multi-specialist clinical consultation by modeling multiple expert perspectives within a single neural network.
• To improve report quality and diversity by enabling expert tokens to learn complementary visual representations through orthogonal regularization.
• To develop a computationally efficient alternative to ensemble models that maintains high performance with minimal parameter overhead.

Proposed method

• Introduces multiple learnable 'expert tokens' into both the encoder and decoder of a transformer backbone, enabling parallel expert modeling.
• In the encoder, expert tokens attend to visual tokens and interact with each other via linear and bilinear attention mechanisms to capture diverse image regions.
• Applies an orthogonal loss to minimize overlap between expert tokens, encouraging them to attend to distinct, complementary image regions.
• In the decoder, each expert token guides cross-attention between word embeddings and visual tokens, producing M candidate reports for M experts.
• Employs a metrics-based expert voting strategy to select the final report from M candidates, improving robustness and clinical relevance.
• Uses a two-stream architecture: vision transformer (ViT) encoder and bilinear transformer encoder to enhance fine-grained feature learning.

Experimental results

Research questions

• RQ1 Can multiple learnable expert tokens in a transformer-based model improve radiology report generation by enabling diverse and complementary attention to image regions?
• RQ2 How does the orthogonal regularization of expert tokens affect attention diversity and report quality compared to single-expert models?
• RQ3 Can the proposed multi-expert framework achieve better performance than ensemble methods while using significantly fewer parameters?
• RQ4 To what extent does the metrics-based expert voting strategy improve the clinical relevance and accuracy of generated reports?
• RQ5 Does the model scale efficiently with increasing numbers of expert tokens without performance degradation?

Key findings

• METransformer achieves state-of-the-art performance on both IU-Xray and MIMIC-CXR benchmarks, outperforming existing single-expert and ensemble-based methods.
• Increasing the number of expert tokens from 1 to 7 significantly improves CIDEr scores, with the highest performance observed at 7 experts.
• When the number of experts exceeds 7 (e.g., 9), performance declines slightly, suggesting an optimal balance between diversity and noise from irrelevant attention.
• METransformer with 7 expert tokens achieves better results than three ensemble baselines while using only 0.007M (0.05‰) more parameters than a single-expert model.
• Qualitative analysis confirms that each expert token focuses on a distinct, clinically relevant image region, such as the rib-diaphragm angle, improving localization of key findings.
• Visual attention maps show that METransformer better aligns generated text with relevant anatomical structures, such as the heart, compared to the baseline model.