[Paper Review] A Survey of Multimodal Information Fusion for Smart Healthcare: Mapping the Journey from Data to Wisdom

Thanveer Shaik, Xiaohui Tao+3 more|arXiv (Cornell University)|Jun 21, 2023

Artificial Intelligence in Healthcare8 citations

TL;DR

This survey maps DIKW-based multimodal fusion for smart healthcare, review techniques, datasets, challenges, and a generic framework for future DIKW-aligned fusion.

ABSTRACT

Multimodal medical data fusion has emerged as a transformative approach in smart healthcare, enabling a comprehensive understanding of patient health and personalized treatment plans. In this paper, a journey from data to information to knowledge to wisdom (DIKW) is explored through multimodal fusion for smart healthcare. We present a comprehensive review of multimodal medical data fusion focused on the integration of various data modalities. The review explores different approaches such as feature selection, rule-based systems, machine learning, deep learning, and natural language processing, for fusing and analyzing multimodal data. This paper also highlights the challenges associated with multimodal fusion in healthcare. By synthesizing the reviewed frameworks and theories, it proposes a generic framework for multimodal medical data fusion that aligns with the DIKW model. Moreover, it discusses future directions related to the four pillars of healthcare: Predictive, Preventive, Personalized, and Participatory approaches. The components of the comprehensive survey presented in this paper form the foundation for more successful implementation of multimodal fusion in smart healthcare. Our findings can guide researchers and practitioners in leveraging the power of multimodal fusion with the state-of-the-art approaches to revolutionize healthcare and improve patient outcomes.

Motivation & Objective

• Adopt and adapt the DIKW model to multimodal fusion in smart healthcare.
• Survey data modalities and fusion approaches from data to wisdom.
• Provide a taxonomy and a generic DIKW-aligned fusion framework for future work.
• Identify challenges and propose solutions to guide research and practice.

Proposed method

• Describe data modalities in smart healthcare (EHRs, imaging, wearables, genomics, sensors, environment, behavior).
• Review state-of-the-art fusion techniques across feature selection, rule-based systems, ML, deep learning, and NLP.
• Propose a DIKW-consistent taxonomy and a generic fusion framework.
• Synthesize challenges, trends, and future directions aligned with Predictive, Preventive, Personalised, and Participatory (4Ps) healthcare.

Experimental results

Research questions

• RQ1 What modalities are used in multimodal fusion for smart healthcare and how are they represented?
• RQ2 What are the main methodological approaches for fusing multimodal medical data?
• RQ3 How can a DIKW-based framework organize current techniques and guide future research?
• RQ4 What challenges and future directions emerge for DIKW-aligned multimodal fusion in healthcare?

Key findings

• DIKW-based representation (Data, Information, Knowledge, Wisdom) provides a cyclical, feedback-enabled view of fusion in smart healthcare.
• A taxonomy links modalities and fusion techniques (feature selection, rule-based, ML, deep learning, NLP) within the DIKW framework.
• A generic DIKW-aligned multimodal fusion framework is proposed to guide future research and practical deployment.
• Substantial challenges exist in data quality, privacy, security, clinical integration, ethics, and interpretation of results.
• A broad set of datasets and modalities (EHRs, imaging, wearables, genomics, sensors, environment, behavior) support multimodal fusion research.
• Future directions emphasize the 4Ps of healthcare (Predictive, Preventive, Personalised, Participatory).