[Paper Review] EPIE Dataset: A Corpus For Possible Idiomatic Expressions

Prateek Saxena, Soma Paul|arXiv (Cornell University)|Jun 16, 2020

Natural Language Processing Techniques14 references4 citations

TL;DR

This paper introduces the EPIE dataset, a large-scale, semi-automatically constructed corpus of 25,206 English sentences annotated with lexical instances of 717 idiomatic expressions, categorized into Static and Formal types. The dataset enables high-accuracy sequence labeling for idiomatic expression detection using a BiLSTM-CRF model, achieving up to 95% precision and 92% recall on independent test sets, significantly advancing metaphor and idiom detection in NLP.

ABSTRACT

Idiomatic expressions have always been a bottleneck for language comprehension and natural language understanding, specifically for tasks like Machine Translation(MT). MT systems predominantly produce literal translations of idiomatic expressions as they do not exhibit generic and linguistically deterministic patterns which can be exploited for comprehension of the non-compositional meaning of the expressions. These expressions occur in parallel corpora used for training, but due to the comparatively high occurrences of the constituent words of idiomatic expressions in literal context, the idiomatic meaning gets overpowered by the compositional meaning of the expression. State of the art Metaphor Detection Systems are able to detect non-compositional usage at word level but miss out on idiosyncratic phrasal idiomatic expressions. This creates a dire need for a dataset with a wider coverage and higher occurrence of commonly occurring idiomatic expressions, the spans of which can be used for Metaphor Detection. With this in mind, we present our English Possible Idiomatic Expressions(EPIE) corpus containing 25206 sentences labelled with lexical instances of 717 idiomatic expressions. These spans also cover literal usages for the given set of idiomatic expressions. We also present the utility of our dataset by using it to train a sequence labelling module and testing on three independent datasets with high accuracy, precision and recall scores.

Motivation & Objective

• To address the lack of large-scale, high-coverage datasets for detecting idiomatic expressions in natural language.
• To overcome the limitations of existing datasets, which are small and limited in coverage, especially for phrasal idioms.
• To enable machine learning-based detection of idiomatic expressions by providing a rich, structured dataset with both literal and idiomatic usages.
• To distinguish between Static and Formal idioms, requiring different detection strategies due to morphosyntactic variation.
• To support downstream NLP tasks such as machine translation and metaphor detection by improving idiomatic expression recognition.

Proposed method

• The dataset is constructed from the British National Corpus (BNC) using StringNet to extract candidate idiomatic instances based on surface forms.
• Candidate idioms are sourced from the IMIL dataset, which maps 2,000 high-frequency English idioms to Indian languages.
• A two-stage filtering process removes non-idiomatic uses and noisy patterns, retaining only sentences with plausible idiomatic spans.
• The dataset is split into Static idioms (exact string matches) and Formal idioms (inflected, modified forms), requiring distinct detection approaches.
• A BiLSTM-CRF sequence labeling model is trained on the Formal idiom subset using 300D GloVe embeddings and evaluated on three independent datasets.
• The model is fine-tuned and tested on SemEval-2013 Task 5b and PIE Corpus, with performance measured on both all usages and only idiomatic usages.

Experimental results

Research questions

• RQ1 Can a large-scale, semi-automatically constructed dataset improve the detection of idiomatic expressions in English text?
• RQ2 How does the distinction between Static and Formal idioms affect the performance of sequence labeling models in idiomatic expression detection?
• RQ3 To what extent can a model trained on the EPIE dataset generalize to independent, out-of-domain datasets such as SemEval-2013 and PIE Corpus?
• RQ4 What is the distribution of idiomatic expression occurrences across the dataset, and how does it affect model learning?
• RQ5 Can a sequence labeling approach achieve high precision and recall for detecting both literal and idiomatic uses of phrasal idioms?

Key findings

• The EPIE dataset contains 25,206 sentences with 717 idiomatic expressions, of which 21,891 contain Static idioms and 3,135 contain Formal idioms.
• The model trained on the EPIE Formal idiom subset achieved 98% accuracy, 95% precision, and 91% recall on its own test split.
• On the SemEval-2013 "all words" dataset, the model achieved 84% accuracy, 90% precision, and 85% recall for all usages, and 86% accuracy, 93% precision, and 86% recall for only idiomatic usages.
• On the SemEval-2013 "lex sample" dataset, the model achieved 89% accuracy and 90% precision for all usages, and 92% accuracy, 95% precision, and 92% recall for idiomatic usages.
• On the PIE Corpus, the model achieved 69% accuracy and 60% precision for all usages, but 88% accuracy, 94% precision, and 88% recall for only idiomatic usages.
• The mean number of occurrences per Formal idiom is 8.75 with a standard deviation of 8.61, indicating an exponential-like distribution, while Static idioms show a highly skewed distribution with a mean of 60.9 and standard deviation of 160.