[Paper Review] Gaussian Smoothen Semantic Features (GSSF) -- Exploring the Linguistic Aspects of Visual Captioning in Indian Languages (Bengali) Using MSCOCO Framework

Chiranjib Sur|arXiv (Cornell University)|Feb 16, 2020

Multimodal Machine Learning Applications72 references4 citations

TL;DR

This paper proposes Gaussian Smoothen Semantic Features (GSSF) to enhance visual captioning in Bengali, an Indian language with complex grammar, by refining semantic representations through Gaussian smoothing of sparse semantic tensors. Using translated English captions from the MSCOCO dataset and fine-tuned LSTM-based models, GSSF improves caption quality—achieving a 4% BLEU-4 gain over baseline LSTM and 1% over GSSCN-LSTM—demonstrating its effectiveness in low-resource, unstructured language settings.

ABSTRACT

In this work, we have introduced Gaussian Smoothen Semantic Features (GSSF) for Better Semantic Selection for Indian regional language-based image captioning and introduced a procedure where we used the existing translation and English crowd-sourced sentences for training. We have shown that this architecture is a promising alternative source, where there is a crunch in resources. Our main contribution of this work is the development of deep learning architectures for the Bengali language (is the fifth widely spoken language in the world) with a completely different grammar and language attributes. We have shown that these are working well for complex applications like language generation from image contexts and can diversify the representation through introducing constraints, more extensive features, and unique feature spaces. We also established that we could achieve absolute precision and diversity when we use smoothened semantic tensor with the traditional LSTM and feature decomposition networks. With better learning architecture, we succeeded in establishing an automated algorithm and assessment procedure that can help in the evaluation of competent applications without the requirement for expertise and human intervention.

Motivation & Objective

• Address the scarcity of annotated visual captioning datasets for Indian regional languages like Bengali, which have complex grammatical structures distinct from English.
• Explore the viability of machine translation as a data augmentation strategy for training visual captioning models in low-resource Indian languages.
• Develop a novel deep learning architecture that enhances semantic representation for image captioning by smoothing sparse semantic tensors using Gaussian techniques.
• Improve caption quality through better semantic feature selection and representation, particularly for unstructured languages with non-linear syntax.
• Establish an automated, expert-free evaluation framework for assessing image captioning performance in non-English, low-resource languages.

Proposed method

• Utilize the Google Translate API (googletrans.Translator) to generate Bengali captions from English reference captions in the MSCOCO dataset, creating a synthetic training corpus.
• Apply Gaussian Smoothen Semantic Features (GSSF) to refine sparse semantic tensors, transforming them into denser, more informative representations by applying Gaussian smoothing to diagonal matrices.
• Integrate GSSF into two novel LSTM-based architectures: GST-LSTM (Gaussian Smoothened Temporal LSTM) and GSSCN-LSTM (Gaussian Smoothened Semantic Convolutional Network-LSTM), enhancing feature learning.
• Employ pre-trained Stanford GloVe word embeddings for Bengali-English mapping, with gradual smoothing to improve semantic consistency and reduce sparsity.
• Train and evaluate models using standard metrics including BLEU-4, BLEU-1, CIDEr-D, and ROUGE-L, comparing performance against a standard LSTM baseline.
• Conduct qualitative analysis on generated captions to assess linguistic correctness, contextual relevance, and descriptive richness in Bengali.

Experimental results

Research questions

• RQ1 Can machine translation be effectively leveraged as a data generation strategy for training visual captioning models in low-resource Indian languages like Bengali?
• RQ2 How does Gaussian smoothing of semantic tensors improve the quality and diversity of generated captions in Bengali image captioning?
• RQ3 To what extent does the proposed GSSF-based architecture outperform standard LSTM and other feature refinement methods in terms of BLEU, CIDEr-D, and ROUGE-L metrics?
• RQ4 Can the GSSF method effectively handle the grammatical and syntactic complexity of unstructured Indian languages such as Bengali, which differ significantly from English?
• RQ5 How do the qualitative outputs of the model compare in terms of linguistic coherence and contextual accuracy to human-annotated captions?

Key findings

• The GST-LSTM model achieved a BLEU-4 score of 0.35 on original Bengali generated captions, representing a 4% improvement over the baseline LSTM (0.31) and a 1% improvement over GSSCN-LSTM (0.34).
• On English-translated generated captions, GST-LSTM achieved a CIDEr-D score of 0.80, outperforming the baseline LSTM (0.62) and GSSCN-LSTM (0.78), indicating better alignment with reference captions.
• The GSSF method significantly enhanced semantic representation by smoothing sparse semantic tensors, leading to more diverse and contextually accurate caption generation.
• Qualitative analysis confirmed that GST-LSTM generated captions were more descriptive, grammatically correct, and contextually relevant than baseline models.
• The GSSF approach demonstrated robustness in handling the morphosyntactic complexity of Bengali, a language with high Sanskrit-derived vocabulary and non-linear word order.
• The study establishes a viable, automated, and low-cost pipeline for training visual captioning models in low-resource Indian languages using translation-based data and semantic feature smoothing.