[Paper Review] Learning to parse from a semantic objective: It works. Is it syntax?

Adina Williams, Andrew Drozdov+1 more|arXiv (Cornell University)|Sep 4, 2017

Natural Language Processing Techniques25 references12 citations

TL;DR

This paper investigates whether neural models trained to parse sentences via a semantic objective learn meaningful syntax. Despite outperforming conventional parsers on sentence classification, the models produce shallow, inconsistent parses that do not align with standard syntactic or semantic formalisms.

ABSTRACT

Recent work on the problem of latent tree learning has made it possible to train neural networks that learn to both parse a sentence and use the resulting parse to interpret the sentence, all without exposure to ground-truth parse trees at training time. Surprisingly, these models often perform better at sentence understanding tasks than models that use parse trees from conventional parsers. This paper aims to investigate what these latent tree learning models learn. We replicate two such models in a shared codebase and find that (i) only one of these models outperforms conventional tree-structured models on sentence classification, (ii) its parsing strategies are not especially consistent across random restarts, (iii) the parses it produces tend to be shallower than standard Penn Treebank (PTB) parses, and (iv) they do not resemble those of PTB or any other semantic or syntactic formalism that the authors are aware of.

Motivation & Objective

• To investigate whether latent tree learning models trained without gold parse trees learn meaningful syntactic structures.
• To compare the performance of such models against conventional tree-structured models on sentence classification.
• To analyze the consistency and structural properties of parses generated by these models across random restarts.
• To evaluate whether the learned parses resemble any known syntactic or semantic formalism, such as Penn Treebank parses.

Proposed method

• Replicated two state-of-the-art latent tree learning models in a shared codebase for direct comparison.
• Trained models end-to-end using a semantic objective without access to gold-standard parse trees during training.
• Evaluated model performance on sentence classification to assess utility for downstream understanding tasks.
• Analyzed parse structures for consistency across multiple random weight initializations and training runs.
• Compared the depth and structure of generated parses against standard Penn Treebank (PTB) parses.
• Assessed whether the learned parses resemble any known syntactic or semantic formalism.

Experimental results

Research questions

• RQ1 Do models trained via a semantic objective learn syntactically meaningful parse structures?
• RQ2 How do the parses produced by these models compare in depth and structure to standard Penn Treebank parses?
• RQ3 Are the parsing strategies of these models consistent across different random weight initializations?
• RQ4 Do the learned parses align with any known semantic or syntactic formalism?
• RQ5 Why do these models outperform conventional parsers on sentence classification despite lacking explicit syntactic supervision?

Key findings

• Only one of the two replicated models outperformed conventional tree-structured models on sentence classification.
• The top-performing model exhibited inconsistent parsing strategies across random restarts, indicating unstable learning.
• The parses generated by the best-performing model were significantly shallower than standard Penn Treebank parses.
• The structure of the learned parses did not resemble any known syntactic or semantic formalism, including PTB or other established frameworks.
• Despite lacking explicit syntactic supervision, the model achieved strong performance on sentence understanding tasks, suggesting it may be learning indirect or non-syntactic representations.