[Paper Review] Ultrasound Video Summarization using Deep Reinforcement Learning

Tianrui Liu, Qingjie Meng+4 more|arXiv (Cornell University)|May 19, 2020

Video Analysis and Summarization18 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based video summarization method for ultrasound videos, using a bi-directional LSTM to model frame features and a reward function emphasizing diagnostic view plane detection, representativeness, and diversity. The approach achieves state-of-the-art performance, especially in unsupervised settings, with a 63.23% F1-score on fetal ultrasound summaries, preserving critical diagnostic information while reducing video length by up to 85%.

ABSTRACT

Video is an essential imaging modality for diagnostics, e.g. in ultrasound imaging, for endoscopy, or movement assessment. However, video hasn't received a lot of attention in the medical image analysis community. In the clinical practice, it is challenging to utilise raw diagnostic video data efficiently as video data takes a long time to process, annotate or audit. In this paper we introduce a novel, fully automatic video summarization method that is tailored to the needs of medical video data. Our approach is framed as reinforcement learning problem and produces agents focusing on the preservation of important diagnostic information. We evaluate our method on videos from fetal ultrasound screening, where commonly only a small amount of the recorded data is used diagnostically. We show that our method is superior to alternative video summarization methods and that it preserves essential information required by clinical diagnostic standards.

Motivation & Objective

• To address the challenge of inefficient retrospective analysis of long ultrasound video recordings in clinical practice.
• To develop a fully automatic video summarization method that preserves diagnostically essential information without relying on human-annotated labels.
• To frame ultrasound video summarization as a reinforcement learning decision-making task to optimize frame selection.
• To introduce a novel diagnostic view plane reward that improves summarization quality by aligning with clinical standards.
• To evaluate the method on fetal ultrasound screening videos, where only a small fraction of data is typically used diagnostically.

Proposed method

• The method uses a CNN-based encoder to extract deep visual features from input video frames, followed by a bi-directional LSTM for sequential modeling of temporal dependencies.
• A reinforcement learning agent selects key frames for the summary by predicting actions (include or exclude) based on a composite reward function.
• The reward function combines three components: representativeness ($\mathcal{R}_{\text{rep}}$), diversity ($\mathcal{R}_{\text{div}}$), and diagnostic view plane likelihood ($\mathcal{R}_{\text{det}}$).
• The model is trained using policy gradient methods, with the objective of maximizing expected cumulative reward over frame sequences.
• The framework supports both supervised and unsupervised training, enabling deployment even without clinical annotations.
• The diagnostic view plane reward is derived from a pre-trained standard plane detection network, aligning frame selection with clinically relevant views.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning framework effectively summarize ultrasound videos while preserving diagnostically critical content?
• RQ2 How does incorporating a diagnostic view plane reward improve summarization performance compared to standard representativeness and diversity rewards?
• RQ3 To what extent can the method perform well in an unsupervised setting without clinical annotations?
• RQ4 How does summary length affect the quality of the generated summaries?
• RQ5 Does the proposed method outperform existing video summarization baselines on real clinical ultrasound data?

Key findings

• The proposed method achieves a 63.23% F1-score on the supervised setting and 59.58% on the unsupervised setting when summarizing videos to 15% of their original length.
• The inclusion of the diagnostic view plane reward ($\mathcal{R}_{\text{det}}$) alone improves unsupervised F1-score by 11.45 points compared to using only $\mathcal{R}_{\text{rep}}$ and $\mathcal{R}_{\text{div}}$.
• When all three reward components are combined, the model achieves the highest performance, with a 63.23% F1-score in the supervised case and 59.58% in the unsupervised case.
• As summary length increases (from 15% to 45%), F1 scores consistently improve across both supervised and unsupervised models, indicating scalability.
• Qualitative results show higher overlap with ground-truth summaries compared to a prior SOTA method, demonstrating better preservation of essential diagnostic frames.
• The ablation study confirms that $\mathcal{R}_{\text{det}}$ is the most impactful component, especially in low-resource (unsupervised) scenarios.