[Paper Review] Dual-stream Maximum Self-attention Multi-instance Learning

Bin Li, Kevin W. Eliceiri|arXiv (Cornell University)|Jun 9, 2020

Image Retrieval and Classification Techniques22 references4 citations

TL;DR

This paper proposes Dual-stream Maximum Self-attention Multi-instance Learning (DSMIL), a novel neural network-based MIL model that jointly learns instance and bag classifiers using a dual-stream architecture. The first stream uses max-pooling to identify the top-activated instance, which then serves as a query to compute self-attention scores across all instances in the second stream, reducing computational complexity from O(n²) to O(n) while improving performance, achieving state-of-the-art results on benchmark MIL datasets.

ABSTRACT

Multi-instance learning (MIL) is a form of weakly supervised learning where a single class label is assigned to a bag of instances while the instance-level labels are not available. Training classifiers to accurately determine the bag label and instance labels is a challenging but critical task in many practical scenarios, such as computational histopathology. Recently, MIL models fully parameterized by neural networks have become popular due to the high flexibility and superior performance. Most of these models rely on attention mechanisms that assign attention scores across the instance embeddings in a bag and produce the bag embedding using an aggregation operator. In this paper, we proposed a dual-stream maximum self-attention MIL model (DSMIL) parameterized by neural networks. The first stream deploys a simple MIL max-pooling while the top-activated instance embedding is determined and used to obtain self-attention scores across instance embeddings in the second stream. Different from most of the previous methods, the proposed model jointly learns an instance classifier and a bag classifier based on the same instance embeddings. The experiments results show that our method achieves superior performance compared to the best MIL methods and demonstrates state-of-the-art performance on benchmark MIL datasets.

Motivation & Objective

• To address the challenge of weakly supervised learning in multi-instance learning (MIL) where instance-level labels are unavailable.
• To reduce the computational complexity of full self-attention in MIL models, which scales quadratically with bag size.
• To improve classification performance by focusing attention on the most relevant instance (top-activated) for dependency modeling.
• To enable joint training of instance and bag classifiers using shared instance embeddings, enhancing feature learning.
• To support training on large-scale bags, such as whole slide images with tens of thousands of patches, by enabling alternating optimization between streams.

Proposed method

• The model employs a dual-stream architecture: the first stream performs standard max-pooling to identify the top-activated instance embedding.
• The second stream computes self-attention scores by correlating only the top-activated instance with all other instances in the bag, forming a query-key mechanism.
• This top-activated self-attention mechanism reduces computation complexity from O(n²) to O(n), making it scalable for large bags.
• The attention scores are used to aggregate instance embeddings into a bag embedding, which is then used for bag-level classification.
• The model jointly trains an instance classifier (from the max-pooling stream) and a bag classifier (from the attention stream), sharing the same instance embeddings.
• Training alternates between the two streams, enabling effective backpropagation through deep feature extractors even on very large bags.

Experimental results

Research questions

• RQ1 Can focusing self-attention on only the top-activated instance improve performance and reduce computational cost in MIL models?
• RQ2 Does a dual-stream architecture that jointly learns instance and bag classifiers lead to better feature representation than single-stream approaches?
• RQ3 Can the proposed method scale to large bags, such as those in whole slide image analysis with tens of thousands of patches?
• RQ4 How does the performance of the instance classifier in DSMIL compare to that in standard MIL max-pooling models?
• RQ5 What is the impact of alternating training between the instance and bag streams on model convergence and performance?

Key findings

• DSMIL achieves state-of-the-art performance on the MIL benchmark datasets, outperforming existing methods including AbMILP, IQSA-AbMILP, and LSA-AbMILP.
• On the colon cancer dataset, DSMIL achieves 91.1% accuracy (±0.012), 93.2% precision (±0.013), and 91.2% F-score (±0.009), surpassing all baselines.
• On the breast cancer dataset, DSMIL achieves 93.7% accuracy (±0.000), 92.5% precision (±0.091), and 97.4% F-score (±0.050), demonstrating high stability and performance.
• The model reduces computational complexity from O(n²) to O(n) by limiting self-attention to only the top-activated instance, enabling scalability to large bags.
• The instance classifier stream can be used independently after training, providing reliable instance-level predictions without requiring full bag attention.
• The alternating training strategy enables effective optimization of deep feature extractors and aggregation operators on extremely large bags, such as those in whole slide image classification.