[Paper Review] IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Chihaya Matsuhira, Marc A. Kastner+5 more|arXiv (Cornell University)|Mar 6, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes IPA-CLIP, a vision-and-language pretraining model that integrates phonetic priors via an IPA-based phoneme embedding to improve understanding of pronunciation similarity, especially for nonwords. By distilling a frozen CLIP text encoder, the model learns a pronunciation encoder that enhances multimodal retrieval and aligns better with human perception of phonetic similarity.

ABSTRACT

Recently, large-scale Vision and Language (V\&L) pretraining has become the standard backbone of many multimedia systems. While it has shown remarkable performance even in unseen situations, it often performs in ways not intuitive to humans. Particularly, they usually do not consider the pronunciation of the input, which humans would utilize to understand language, especially when it comes to unknown words. Thus, this paper inserts phonetic prior into Contrastive Language-Image Pretraining (CLIP), one of the V\&L pretrained models, to make it consider the pronunciation similarity among its pronunciation inputs. To achieve this, we first propose a phoneme embedding that utilizes the phoneme relationships provided by the International Phonetic Alphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIP text encoder, we train a pronunciation encoder employing the IPA-based embedding. The proposed model named IPA-CLIP comprises this pronunciation encoder and the original CLIP encoders (image and text). Quantitative evaluation reveals that the phoneme distribution on the embedding space represents phonetic relationships more accurately when using the proposed phoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirm that the proposed pronunciation encoder enhances the performance of the text encoder and that the pronunciation encoder handles nonsense words in a more phonetic manner than the text encoder. Finally, qualitative evaluation verifies the correlation between the pronunciation encoder and human perception regarding pronunciation similarity.

Motivation & Objective

• To address the limitation of vision-and-language models in considering pronunciation similarity, especially for nonwords or unknown terms.
• To integrate phonetic priors into CLIP by leveraging the International Phonetic Alphabet (IPA) chart to model phoneme relationships.
• To develop a pronunciation encoder that improves multimodal retrieval performance and handles nonwords more robustly than standard text-based CLIP.
• To evaluate whether the learned pronunciation embedding space correlates with human perception of phonetic similarity.

Proposed method

• Propose an IPA-based phoneme embedding that encodes phonetic relationships from the IPA chart into a continuous vector space.
• Train a pronunciation encoder via knowledge distillation from a frozen CLIP text encoder, using the IPA-based embedding as input.
• Extend CLIP with the new pronunciation encoder while keeping the original image and text encoders frozen.
• Use contrastive pretraining on image-text pairs to align the pronunciation encoder’s output with the CLIP’s shared embedding space.
• Evaluate the pronunciation encoder’s performance on multimodal retrieval and human perception of phonetic similarity.
• Apply a rank correlation metric to assess how well the embedding space reflects human-rated pronunciation similarity.

Experimental results

Research questions

• RQ1 Can an IPA-based phoneme embedding better represent phonetic relationships than standard text-based embeddings in a vision-and-language pretraining setup?
• RQ2 Does integrating a pronunciation encoder into CLIP improve performance on multimodal retrieval tasks, especially with nonwords?
• RQ3 How well does the learned pronunciation embedding space correlate with human perception of pronunciation similarity?
• RQ4 Does the proposed method enhance robustness to nonwords compared to standard CLIP?
• RQ5 What are the limitations of the approach in handling short-syllable words with high phonetic ambiguity?

Key findings

• The IPA-based phoneme embedding improves the representation of phonetic relationships in the embedding space, as shown by better alignment with the IPA chart's phonetic structure.
• In multimodal retrieval, IPA-CLIP outperforms standard CLIP, particularly when nonwords are involved, demonstrating enhanced robustness.
• The pronunciation encoder achieves a rank correlation of 0.642 with human perception for the word 'Sit', significantly outperforming CLIP's 0.353.
• For 'Wonder', the proposed method achieves a rank correlation of 0.640, indicating strong alignment with human perception of pronunciation similarity.
• The performance of the frozen version of the proposed model is lower for short words like 'Sit', suggesting sensitivity to phonetic ambiguity in short syllables.
• Overall, all pronunciation-based methods outperform the text-based CLIP baseline in human perception correlation, validating the effectiveness of phonetic priors.