[Paper Review] Asymmetrically Weighted CCA And Hierarchical Kernel Sentence Embedding For Image & Text Retrieval

Youssef Mroueh, Etienne Marcheret+1 more|arXiv (Cornell University)|Nov 19, 2015

Advanced Image and Video Retrieval Techniques32 references3 citations

TL;DR

This paper proposes Asymmetrically Weighted CCA (AW-CCA) for improved bidirectional image and text retrieval by mapping search items to the query space using asymmetric canonical weight scaling, combined with a computationally efficient T-SVD-guided Tikhonov cross-validation for regularization and a novel Hierarchical Kernel Sentence Embedding (HKSE) for superior sentence representation. The method achieves state-of-the-art performance on MSCOCO and Flickr benchmarks using off-the-shelf features.

ABSTRACT

Joint modeling of language and vision has been drawing increasing interest. A multimodal data representation allowing for bidirectional retrieval of images by sentences and vice versa is a key aspect. In this paper we present three contributions in canonical correlation analysis (CCA) based multimodal retrieval. Firstly, we show that an asymmetric weighting of the canonical weights, while achieving a cross view mapping from the search to the query space, improves the retrieval performance. Secondly, we devise a computationally efficient model selection, crucial to generalization and stability, in the framework of the Bj\"ork Golub algorithm for regularized CCA via spectral filtering. Finally, we introduce a Hierarchical Kernel Sentence Embedding (HKSE) that approximates Kernel CCA for a special similarity kernel between distribution of words embedded in a vector space. State of the art results are obtained on MSCOCO and Flickr benchmarks when these three techniques are used in conjunction.

Motivation & Objective

• To improve bidirectional image and text retrieval by rethinking the direction of cross-view mapping in CCA.
• To enhance model generalization and numerical stability through efficient regularization and model selection in regularized CCA.
• To develop a more effective sentence embedding method that generalizes and outperforms mean word2vec for multimodal retrieval.
• To achieve state-of-the-art retrieval performance on standard benchmarks using only off-the-shelf image and text features.

Proposed method

• AW-CCA maps search space items to the query space by asymmetrically weighting the canonical correlation weights using the canonical correlations, improving retrieval performance.
• Regularized CCA is implemented via spectral filtering using the Björk-Golub algorithm, with two regularization families: Tikhonov and truncated SVD (T-SVD).
• A hybrid cross-validation strategy is proposed that leverages the fast computation of T-SVD regularization to guide Tikhonov regularization path selection, enabling efficient and stable model selection.
• Hierarchical Kernel Sentence Embedding (HKSE) aggregates word embeddings (e.g., word2vec) using a hierarchical kernel approach that models word distribution similarity in vector space, outperforming simple averaging.
• The final system combines AW-CCA with HKSE and cosine similarity for cross-modal matching, enabling end-to-end bidirectional retrieval.
• Model selection is performed using a validation set, and the method is evaluated on MSCOCO and Flickr benchmarks with standard metrics (r@1, r@5, r@10, median rank).

Experimental results

Research questions

• RQ1 Does mapping search items to the query space via asymmetric weighting of CCA weights improve bidirectional retrieval performance?
• RQ2 Can T-SVD regularization be effectively used in CCA with a computationally efficient cross-validation strategy for model selection?
• RQ3 Can a hierarchical kernel-based sentence embedding method outperform mean word2vec in multimodal retrieval tasks?
• RQ4 Does the combination of AW-CCA, HKSE, and efficient regularization achieve state-of-the-art performance on standard image-text retrieval benchmarks?

Key findings

• AW-CCA with asymmetric weighting significantly improves both image search and image captioning retrieval performance compared to standard CCA and cross-view mapping methods.
• The proposed T-SVD-guided Tikhonov cross-validation enables fast and stable model selection, improving generalization with minimal computational overhead.
• HKSE outperforms mean word2vec as a sentence embedding method, providing a more effective aggregation of word-level representations.
• On the MSCOCO benchmark with 1K test images, the best configuration (AW-G-Tikh-CCA + ResNet + [HKSE (lin,rbf)||HKSE(rbf,rbf)]) achieves 39.68% r@1 and 55.16% r@1 for image search and image annotation, respectively.
• On the Flickr 30K benchmark, the same configuration achieves 31.48% r@1 and 43.83% r@1 for image search and image annotation, respectively.
• The method achieves state-of-the-art results on both MSCOCO and Flickr benchmarks without requiring fine-tuned features, using only off-the-shelf image and text embeddings.