[Paper Review] A Framework for (Under)specifying Dependency Syntax without Overloading Annotators

Nathan Schneider, Brendan O’Connor+6 more|arXiv (Cornell University)|Jun 10, 2013

Data Quality and Management25 references12 citations

TL;DR

This paper introduces FUDG (Fragmentary Unlabeled Dependency Grammar), a lightweight dependency syntax annotation framework that allows annotators to underspecify syntactic structures to reduce cognitive load and annotation cost. Using a simple ASCII notation (GFL), the framework supports partial annotations, and evaluation algorithms are developed to measure inter-annotator agreement on underspecified data. The approach enables rapid, low-cost annotation across languages and tasks, with demonstrated success in English, Malagasy, and Kinyarwanda.

ABSTRACT

We introduce a framework for lightweight dependency syntax annotation. Our formalism builds upon the typical representation for unlabeled dependencies, permitting a simple notation and annotation workflow. Moreover, the formalism encourages annotators to underspecify parts of the syntax if doing so would streamline the annotation process. We demonstrate the efficacy of this annotation on three languages and develop algorithms to evaluate and compare underspecified annotations.

Motivation & Objective

• To reduce the cost and complexity of syntactic annotation by allowing annotators to underspecify ambiguous or irrelevant syntactic structures.
• To develop a simple, human-readable notation (GFL) that enables text-based annotation without specialized tools.
• To create algorithms for evaluating inter-annotator agreement on partially specified dependency structures.
• To demonstrate the framework’s feasibility and effectiveness across diverse languages and text types, including social media and dialogue.
• To support scalable, project-specific syntactic annotation by enabling specialization and reducing training burden.

Proposed method

• Proposes FUDG, a formalism extending unlabeled dependencies with support for multiword units, labeled coordination, and anaphora-antecedent relations.
• Introduces Graph Fragment Language (GFL), a lightweight ASCII notation for authoring annotations in any text editor.
• Designs validation, normalization, and visualization tools to support GFL annotation workflows.
• Enables underspecification via fudge expressions (e.g., ~) to represent uncertainty or intentional omission of dependencies.
• Develops new evaluation metrics—promiscuity, commitment, softComPrec—for measuring agreement on partial annotations.
• Applies the framework to three languages (English, Malagasy, Kinyarwanda) and three domains (social media, news, dialogue), using independent and specialized annotators.

Experimental results

Research questions

• RQ1 Can a dependency annotation framework support underspecification without overburdening annotators?
• RQ2 How can inter-annotator agreement be meaningfully measured on partially specified syntactic annotations?
• RQ3 To what extent can specialized annotators produce complementary, high-quality partial annotations?
• RQ4 How does the framework perform across low-resource and informal language genres?
• RQ5 Can the GFL notation and tools enable efficient, scalable annotation with minimal training?

Key findings

• The framework enabled rapid annotation with minimal training, as evidenced by high inter-annotator agreement (F1 up to .95) on tweets and reviews.
• Specialized annotation by two annotators on noun phrase and clause structure produced complementary results, with 95% agreement on the combined fragments (F1 = .95).
• The combination of specialized annotations (C ∩ D) showed higher promiscuity (supporting more possible parses) but still achieved strong softComPrec scores (e.g., .82 on tweets), indicating high consistency.
• Agreement with converted Penn Treebank parses was moderate to high, with softComPrec scores of .91–.93 for the best comparisons, despite the Treebank’s full commitment to single parses.
• The framework successfully supported annotation in three low-resource languages (English, Malagasy, Kinyarwanda), demonstrating cross-linguistic feasibility.
• Pilot crowdsourcing on Mechanical Turk showed that 80 workers with sentence diagramming experience could produce FUDG annotations, suggesting viability for non-expert annotation.