[Paper Review] From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

Charles Zhang, Benji Peng+15 more|arXiv (Cornell University)|Nov 6, 2024

Natural Language Processing Techniques4 citations

TL;DR

This survey reviews the evolution of word embeddings to multimodal embeddings in large language models, covering foundational techniques like Word2Vec and GloVe, contextual models such as BERT and GPT, and advancements in multimodal, cross-lingual, and personalized representations. It identifies key challenges in interpretability, bias, and efficiency, and outlines future research directions toward scalable, grounded, and cognitively plausible models.

ABSTRACT

Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models.

Motivation & Objective

• To provide a comprehensive review of the progression from sparse word representations to dense, contextualized, and multimodal embeddings in NLP.
• To analyze technical advancements in word embeddings, including subword modeling, cross-lingual transfer, and personalized representations.
• To examine challenges in interpretability, bias, and model efficiency, and to identify open problems in embedding-based language modeling.
• To explore the integration of embeddings with vision, robotics, and knowledge bases for more grounded and reasoning-capable AI systems.
• To guide future research by identifying key directions in scalability, interpretability, and cognitive plausibility of embedding models.

Proposed method

• Traces the evolution from one-hot encoding to dense word embeddings (Word2Vec, GloVe, fastText) based on the distributional hypothesis and contextual similarity.
• Reviews contextualized models such as ELMo, BERT, GPT, and XLNet, which generate dynamic representations based on surrounding context.
• Examines subword-level embeddings (e.g., byte-pair encoding) to improve generalization for rare and out-of-vocabulary words.
• Analyzes cross-lingual and multilingual embeddings for zero-shot and few-shot transfer learning across languages.
• Investigates personalized embeddings that model individual linguistic variation and preferences.
• Explores multimodal extensions integrating vision, language, and robotics, including visual grounding and embodied AI.

Experimental results

Research questions

• RQ1 How have word embeddings evolved from static, distributional representations to dynamic, context-aware models?
• RQ2 What are the key technical and architectural innovations enabling contextualized embeddings like BERT and GPT?
• RQ3 How can embeddings be extended to support cross-lingual and multilingual understanding?
• RQ4 What are the main challenges in interpretability, bias, and efficiency in modern embedding models?
• RQ5 What future directions are most critical for grounding embeddings in real-world knowledge and cognitive processes?

Key findings

• Contextualized embeddings such as BERT and GPT significantly outperform static embeddings in capturing polysemy and long-range dependencies.
• Subword-level modeling improves generalization for rare and unseen words, especially in morphologically rich languages.
• Cross-lingual embeddings enable zero-shot transfer learning, reducing the need for parallel corpora in multilingual NLP.
• Personalized embeddings can model individual linguistic variation, enabling tailored language applications like adaptive tutoring.
• Multimodal embeddings grounded in vision and robotics enhance language understanding by linking linguistic representations to sensory and motor experiences.
• Current models still face challenges in interpretability, bias mitigation, and efficient deployment, highlighting the need for scalable and transparent architectures.