[Paper Review] Three-Stream Convolutional Networks for Video-based Person Re-Identification

Yu Zeng, Tianrui Li+4 more|arXiv (Cornell University)|Nov 22, 2017

Video Surveillance and Tracking Methods19 references3 citations

TL;DR

This paper proposes Three-Stream Convolutional Networks (TSCN), a multi-stream architecture that enhances video-based person re-identification by learning complementary features through distinct pooling strategies (max, average, and stride-2 convolution) across three streams. By fusing these streams and incorporating multi-scale and upsampling strategies, TSCN achieves state-of-the-art performance on iLIDS-VID, PRID-2011, and MARS datasets, outperforming prior models by up to 2.6% in rank-5 accuracy.

ABSTRACT

This paper aims to develop a new architecture that can make full use of the feature maps of convolutional networks. To this end, we study a number of methods for video-based person re-identification and make the following findings: 1) Max-pooling only focuses on the maximum value of a receptive field, wasting a lot of information. 2) Networks with different streams even including the one with the worst performance work better than networks with same streams, where each one has the best performance alone. 3) A full connection layer at the end of convolutional networks is not necessary. Based on these studies, we propose a new convolutional architecture termed Three-Stream Convolutional Networks (TSCN). It first uses different streams to learn different aspects of feature maps for attentive spatio-temporal fusion of video, and then merges them together to study some union features. To further utilize the feature maps, two architectures are designed by using the strategies of multi-scale and upsampling. Comparative experiments on iLIDS-VID, PRID-2011 and MARS datasets illustrate that the proposed architectures are significantly better for feature extraction than the state-of-the-art models.

Motivation & Objective

• To address the limitations of max-pooling and full connection layers in video-based person re-identification networks.
• To explore whether multi-stream architectures with diverse feature learning strategies improve performance over single-stream models.
• To investigate the effectiveness of multi-scale and upsampling techniques in reusing and enhancing learned feature maps.
• To design a network architecture that fully utilizes feature maps without relying on a final fully connected layer.
• To achieve state-of-the-art performance on standard video re-identification benchmarks.

Proposed method

• The TSCN architecture uses three parallel streams, each applying a different feature extraction strategy: max-pooling, average-pooling, and 2-stride convolution, to capture distinct aspects of feature maps.
• The outputs of the three streams are concatenated and fused to learn joint representations that combine complementary features.
• Two enhanced variants—Multi-TSCN and Multi-Two-SCN—are proposed by integrating multi-scale feature extraction and upsampling to improve feature reuse.
• The network avoids a final fully connected layer, relying instead on global average pooling and classification heads for end-to-end training.
• Optical flow features are used as input to capture temporal motion, combined with RGB frames for spatio-temporal modeling.
• The model is trained end-to-end on iLIDS-VID, PRID-2011, and MARS datasets using standard re-identification protocols.

Experimental results

Research questions

• RQ1 Does using multiple streams with different pooling strategies improve feature representation in video-based person re-identification compared to single-stream models?
• RQ2 Can multi-scale and upsampling strategies enhance the reuse of learned feature maps and improve performance?
• RQ3 Is a fully connected layer at the end of the network necessary for optimal performance in this task?
• RQ4 Can a three-stream architecture outperform both single-stream and two-stream models in terms of accuracy and robustness?
• RQ5 How do different stream combinations and fusion strategies affect the final re-identification accuracy across diverse datasets?

Key findings

• The proposed TSCN model achieves a rank-1 accuracy of 66.5% on iLIDS-VID and 79.2% on PRID-2011, outperforming all prior state-of-the-art methods including AMOC with the same optical flow algorithm.
• The Multi-TSCN variant achieves 67.5% rank-1 accuracy on iLIDS-VID and 78.8% on PRID-2011, demonstrating further gains from multi-scale and upsampling strategies.
• On the MARS dataset, the TSCN model achieves 45.6% rank-1 accuracy, surpassing RNN-CNN (40.0%) and ASTPN (44.0%), confirming its robustness on large-scale benchmarks.
• The three-stream architecture improves rank-5 accuracy by 2.6 percentage points over the best single-stream baseline on iLIDS-VID, indicating strong complementarity among streams.
• The ablation study confirms that a fully connected layer is not necessary, as global average pooling yields superior performance with reduced parameter count.
• The model’s performance is robust across datasets with varying challenges, indicating strong generalization and effective feature utilization.