[Paper Review] Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval

Donghuo Zeng, Yi Yu+1 more|arXiv (Cornell University)|Aug 10, 2019

Advanced Image and Video Retrieval Techniques65 references4 citations

TL;DR

This paper proposes a deep triplet neural network with cluster-CCA (TNN-C-CCA) for audio-visual cross-modal retrieval, combining triplet loss with cluster-CCA to learn discriminative joint embeddings by maximizing correlation between audio and video modalities while distinguishing positive and negative pairs. The method outperforms six CCA-based and four state-of-the-art cross-modal retrieval methods on two datasets, achieving a MAP of 74.66% in audio-to-visual retrieval and 73.77% in visual-to-audio retrieval with 10 correlation components.

ABSTRACT

Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio.Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose a novel deep triplet neural network with cluster canonical correlation analysis(TNN-C-CCA), which is an end-to-end supervised learning architecture with audio branch and video branch.We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made: i) a better representation by constructing deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. ii) positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over 5-fold cross-validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show the proposed learning architecture with two branches outperforms existing six CCA-based methods and four state-of-the-art based cross-modal retrieval methods.

Motivation & Objective

• To address the lack of research in audio-visual cross-modal retrieval due to limited paired datasets and semantic information.
• To learn joint embeddings that maximize correlation between audio and visual modalities in a shared subspace.
• To improve representation learning by incorporating both positive and negative pairs using triplet loss.
• To preserve semantic information across all samples within the same category, not just pairwise matches.
• To develop an end-to-end supervised architecture that outperforms existing CCA-based and cross-modal retrieval methods.

Proposed method

• Proposes a deep triplet neural network (TNN) with triplet loss to learn discriminative embeddings by optimizing similarity between positive pairs and dissimilarity between negative pairs.
• Integrates cluster-CCA (C-CCA) to learn joint representations by enforcing one-to-one correspondence between all cross-modal data pairs and maximizing correlation in a shared subspace.
• Uses a two-branch architecture with separate audio and video branches, each processed by pre-trained deep CNNs for feature extraction.
• Applies a margin-based triplet loss function to enforce that the distance between positive pairs is smaller than that between negative pairs by at least a margin.
• Employs a shared projection space where audio and visual features are mapped to maximize canonical correlation, with dimensionality controlled by the number of correlation components.
• Conducts 5-fold cross-validation with hyperparameter tuning on margin and batch size to optimize performance.

Experimental results

Research questions

• RQ1 Can a deep triplet network with cluster-CCA improve audio-visual cross-modal retrieval by jointly modeling pairwise and non-pairwise semantic relationships?
• RQ2 How does the inclusion of negative pairs in triplet loss affect the discriminative power of learned embeddings compared to standard CCA or DCCA?
• RQ3 What is the optimal number of correlation components for cluster-CCA in audio-visual retrieval, and how does it affect performance?
• RQ4 How do hyperparameters like margin and batch size influence the model's MAP performance?
• RQ5 Can the proposed TNN-C-CCA architecture outperform existing CCA-based and state-of-the-art cross-modal retrieval methods on real-world audio-visual datasets?

Key findings

• The proposed TNN-C-CCA model achieves a mean average precision (MAP) of 74.66% in audio-to-visual retrieval and 73.77% in visual-to-audio retrieval when using 10 correlation components.
• The model outperforms six CCA-based methods and four state-of-the-art cross-modal retrieval methods on two different audio-visual datasets.
• Optimal performance is achieved at 400 batches with a margin of 0.5, while performance degrades significantly beyond 800 batches due to overfitting or instability.
• The number of correlation components has a strong impact on performance, with 10 components yielding the highest MAP; performance drops sharply at 50 components.
• The model with a batch size of 30 (700 batches) reduces training time to 3 hours, compared to 32 hours with 300 batches, while maintaining high performance.
• Ablation studies confirm that triplet loss with both positive and negative pairs significantly improves embedding quality compared to models relying only on pairwise correlations.