[Paper Review] An improved neural network model for joint POS tagging and dependency parsing

Dat Quoc Nguyen, Karin Verspoor|arXiv (Cornell University)|Jul 11, 2018

Topic Modeling47 references20 citations

TL;DR

This paper proposes a joint neural network model that integrates a BiLSTM-based POS tagger with the BIST graph-based dependency parser, improving both tagging and parsing performance. On the English Penn Treebank, it achieves state-of-the-art results with 94.51% UAS, 92.87% LAS, and 97.97% POS accuracy, outperforming prior models and demonstrating strong gains in multilingual parsing and downstream applications.

ABSTRACT

We propose a novel neural network model for joint part-of-speech (POS) tagging and dependency parsing. Our model extends the well-known BIST graph-based dependency parser (Kiperwasser and Goldberg, 2016) by incorporating a BiLSTM-based tagging component to produce automatically predicted POS tags for the parser. On the benchmark English Penn treebank, our model obtains strong UAS and LAS scores at 94.51% and 92.87%, respectively, producing 1.5+% absolute improvements to the BIST graph-based parser, and also obtaining a state-of-the-art POS tagging accuracy at 97.97%. Furthermore, experimental results on parsing 61 "big" Universal Dependencies treebanks from raw texts show that our model outperforms the baseline UDPipe (Straka and Straková, 2017) with 0.8% higher average POS tagging score and 3.6% higher average LAS score. In addition, with our model, we also obtain state-of-the-art downstream task scores for biomedical event extraction and opinion analysis applications. Our code is available together with all pre-trained models at: https://github.com/datquocnguyen/jPTDP

Motivation & Objective

• To address error propagation in dependency parsing caused by imperfect automatic POS tagging by jointly learning POS tags and dependency structures.
• To improve parsing performance by integrating a BiLSTM-based tagging component into the BIST graph-based parser architecture.
• To achieve state-of-the-art performance on both intrinsic parsing and POS tagging tasks, as well as downstream NLP applications.
• To demonstrate generalization across languages using raw text in 61 Universal Dependencies treebanks.
• To establish a strong, reusable baseline for joint parsing and tagging in low-resource and biomedical NLP settings.

Proposed method

• The model uses a two-component architecture: a BiLSTM-based tagging component that predicts POS tags from word and character-level embeddings.
• Word representations are formed by concatenating pre-trained GloVe word embeddings with character-level representations from a BiLSTM over character sequences.
• The parsing component uses a separate BiLSTM to encode both input words and predicted POS tags, producing latent representations for arc prediction.
• Dependency arcs are predicted via a multilayer perceptron (MLP) on the latent representations, with another MLP for arc labeling.
• The model is trained end-to-end with cross-entropy loss for POS tagging and structured CRF-like decoding for dependency parsing.
• The joint training process allows the tagging component to benefit from parsing supervision and vice versa, improving both tasks.

Experimental results

Research questions

• RQ1 Can joint learning of POS tagging and dependency parsing improve performance on both tasks compared to independent modeling?
• RQ2 Does integrating a BiLSTM-based tagger into a graph-based parser lead to better parsing accuracy and POS tagging accuracy?
• RQ3 How does the joint model perform on multilingual, raw-text parsing tasks using Universal Dependencies treebanks?
• RQ4 Can the joint model achieve state-of-the-art results in downstream applications such as biomedical event extraction and opinion analysis?
• RQ5 Does higher intrinsic parsing performance always lead to better downstream application performance?

Key findings

• On the English Penn Treebank, the model achieves 94.51% UAS and 92.87% LAS, representing a 1.5+% absolute improvement over the BIST parser.
• The model reaches 97.97% POS tagging accuracy, setting a new state-of-the-art on the benchmark.
• On 61 big Universal Dependencies treebanks, the model outperforms UDPipe with 0.8% higher average POS tagging F1, 3.1% higher UAS, and 3.6% higher LAS.
• In the CoNLL 2018 shared task, the model achieved the highest unofficial F1 scores across three downstream tasks, including biomedical event extraction and opinion analysis.
• Despite using less training data than the top-ranked EPE 2017 system, the model achieved higher F1 scores in event extraction and opinion analysis, indicating that intrinsic parsing quality does not always predict downstream performance.
• The model's code and pre-trained weights are publicly available at https://github.com/datquocnguyen/jPTDP.