[Paper Review] The Best of Both Worlds: Lexical Resources To Improve Low-Resource Part-of-Speech Tagging

Barbara Plank, Sigrid Klerke+1 more|arXiv (Cornell University)|Nov 21, 2018

Natural Language Processing Techniques30 references4 citations

TL;DR

This paper proposes DsDs, a neural part-of-speech tagger that softly integrates symbolic lexical resources with pre-trained word embeddings to improve low-resource cross-lingual tagging. By leveraging dictionary-derived linguistic knowledge in a differentiable manner, the model achieves superior robustness and performance even with limited or noisy training data, outperforming methods relying solely on embeddings or explicit symbolic constraints.

ABSTRACT

In natural language processing, the deep learning revolution has shifted the focus from conventional hand-crafted symbolic representations to dense inputs, which are adequate representations learned automatically from corpora. However, particularly when working with low-resource languages, small amounts of symbolic lexical resources such as user-generated lexicons are often available even when gold-standard corpora are not. Such additional linguistic information is though often neglected, and recent neural approaches to cross-lingual tagging typically rely only on word and subword embeddings. While these representations are effective, our recent work has shown clear benefits of combining the best of both worlds: integrating conventional lexical information improves neural cross-lingual part-of-speech (PoS) tagging. However, little is known on how complementary such additional information is, and to what extent improvements depend on the coverage and quality of these external resources. This paper seeks to fill this gap by providing the first thorough analysis on the contributions of lexical resources for cross-lingual PoS tagging in neural times.

Motivation & Objective

• To investigate how symbolic lexical resources can enhance neural cross-lingual part-of-speech tagging in low-resource settings.
• To analyze the impact of lexicon coverage, quality, and composition on model performance.
• To evaluate whether implicit integration of linguistic knowledge outperforms explicit methods like retrofitting or type constraints.
• To understand how linguistic resources influence internal representations in neural sequence models.
• To provide a systematic evaluation of hybrid neural-symbolic approaches in low-resource NLP.

Proposed method

• The model combines pre-trained word embeddings with a soft, differentiable integration of lexical resources, where dictionary entries inform the model without hard constraints.
• It uses a neural sequence model (LSTM-based) that attends to both word embeddings and lexical features during decoding.
• The approach treats dictionary entries as soft supervision, allowing ambiguous entries (e.g., 'play' as noun/verb) to contribute probabilistically to predictions.
• The method is trained end-to-end, with the linguistic knowledge embedded via a learnable projection layer that modulates the hidden states.
• The model is evaluated under low-resource regimes with as few as 5,000 projected training instances.
• Ablation studies compare performance with and without lexical resources, and with varying lexicon compositions and sizes.

Experimental results

Research questions

• RQ1 How does the integration of symbolic lexical resources affect POS tagging accuracy in low-resource cross-lingual settings?
• RQ2 To what extent does the quality and composition of a lexicon matter compared to its size?
• RQ3 Does implicit use of linguistic knowledge (via soft integration) outperform explicit methods like type constraints or retrofitting?
• RQ4 How do internal representations in the model change when exposed to lexical resources?
• RQ5 Can a model learn effectively from very limited projected training data when augmented with lexical knowledge?

Key findings

• The model achieves significant improvements in POS tagging accuracy by integrating lexical resources, even with only 5,000 projected training instances.
• Composition of the lexicon—particularly the balance of part-of-speech types and coverage of high-frequency words—matters more than its overall size.
• Soft integration of lexical knowledge leads to better performance than explicit constraints or retrofitting, suggesting that implicit supervision is more effective in low-resource settings.
• Probing and similarity analysis show that the model’s internal representations become more sensitive to task-relevant linguistic structure when using lexical resources.
• The model demonstrates greater robustness to noise and data sparsity, indicating that lexical knowledge stabilizes learning in low-resource regimes.
• The results support the hypothesis that neural models can learn beyond the explicit coverage of a lexicon, benefiting from implicit linguistic priors.