[Paper Review] Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Ling Wang, Tiago Luís+6 more|arXiv (Cornell University)|Aug 9, 2015

Topic Modeling28 references132 citations

TL;DR

This paper proposes a character-level compositional model using bidirectional LSTMs to generate word representations without relying on word lookup tables. By composing character vectors through a shared, parameter-efficient architecture, the model achieves state-of-the-art performance in language modeling and part-of-speech tagging—especially in morphologically rich languages—while enabling zero-shot generalization to unseen words with minimal parameters.

ABSTRACT

We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form-function relationship in language, our "composed" word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).

Motivation & Objective

• To address the limitations of traditional word lookup tables that cannot generalize to unseen words and require large parameter counts.
• To model the complex, non-compositional form–function relationship in language using character-level composition.
• To enable effective word representation learning in morphologically rich languages where word forms vary significantly.
• To reduce model size and improve generalization by replacing word-level embeddings with character-level compositional modeling.
• To demonstrate that character-level composition via bidirectional LSTMs can capture both morphological regularities and idiosyncratic lexical patterns.

Proposed method

• The model uses a bidirectional LSTM to process character sequences of each word, producing a context-aware word representation from character-level embeddings.
• Each character type is associated with a learnable vector, and the LSTM parameters jointly encode both morphological and lexical knowledge.
• Word representations are generated by passing the character sequence through the bidirectional LSTM, with the final hidden states combined into a single vector representation.
• The model avoids word lookup tables entirely, relying only on character vectors and a fixed set of LSTM parameters, enabling parameter efficiency.
• The architecture is trained end-to-end on language modeling and POS tagging tasks, with shared parameters across all words.
• The model learns to group semantically and syntactically similar words even when orthographically distant, such as 'October' and 'January'.

Experimental results

Research questions

• RQ1 Can a character-level compositional model outperform traditional word lookup tables in language modeling and POS tagging?
• RQ2 To what extent can a character-based model generalize to unseen words without explicit word-level embeddings?
• RQ3 How effective is the model in capturing morphological regularities in morphologically rich languages like Turkish?
• RQ4 Can the model learn non-compositional form–function relationships despite relying only on character sequences?
• RQ5 Does the absence of a word lookup table lead to better parameter efficiency and generalization?

Key findings

• The model achieves state-of-the-art performance on part-of-speech tagging, including a new best result on the English Penn Treebank.
• Significant performance gains are observed in morphologically rich languages such as Turkish, where the model outperforms baselines by a wide margin.
• The model generates semantically and syntactically similar representations for orthographically distant words, such as 'October' and 'January', demonstrating functional similarity learning.
• Despite using only a fraction of the parameters of traditional word lookup tables, the model matches or exceeds their performance on both language modeling and POS tagging.
• The model generalizes effectively to unseen words, such as 'Frenchification', by composing known character components, demonstrating zero-shot capability.
• The model successfully captures both regular morphological patterns and idiosyncratic lexical forms, showing robustness to non-compositional form–function mappings.