[Paper Review] Measuring semantic complexity

Wlodek Zadrozny|ArXiv.org|May 8, 1995

Natural Language Processing Techniques21 references3 citations

TL;DR

This paper introduces a formal framework for measuring semantic complexity using 'meaning automata'—finite-state machines that model semantic understanding. It applies this method to assess complexity in natural language understanding systems, showing that complex behaviors can emerge from semantically simple components, enabling pre-implementation complexity estimation for NLP systems.

ABSTRACT

We define {\em semantic complexity} using a new concept of {\em meaning automata}. We measure the semantic complexity of understanding of prepositional phrases, of an "in depth understanding system", and of a natural language interface to an on-line calendar. We argue that it is possible to measure some semantic complexities of natural language processing systems before building them, and that systems that exhibit relatively complex behavior can be built from semantically simple components.

Motivation & Objective

• To define a formal, measurable notion of semantic complexity in natural language understanding.
• To develop a method for estimating the semantic complexity of NLP systems before implementation.
• To demonstrate that complex semantic behaviors can be constructed from semantically simple components.
• To apply the framework to real-world systems, including a calendar interface and in-depth understanding systems.
• To provide a theoretical foundation for evaluating and comparing semantic processing systems.

Proposed method

• Introduces 'meaning automata' as formal models of semantic understanding, where states represent semantic states and transitions represent meaning changes.
• Uses automata to represent the semantic behavior of natural language processing components, such as prepositional phrase interpretation.
• Defines semantic complexity as the number of states and transitions in the meaning automaton, reflecting the system's representational complexity.
• Applies the framework to analyze a natural language interface to an online calendar, decomposing its semantic behavior into automaton components.
• Employs a formal, state-based representation to quantify the complexity of understanding beyond syntactic or lexical analysis.
• Proposes that complexity can be measured independently of implementation, enabling early-stage system design evaluation.

Experimental results

Research questions

• RQ1 Can semantic complexity in natural language understanding systems be formally measured before implementation?
• RQ2 How can meaning automata be used to model and quantify the complexity of semantic understanding in NLP systems?
• RQ3 To what extent can complex semantic behaviors be constructed from simpler semantic components?
• RQ4 What is the semantic complexity of a natural language interface to an online calendar system?
• RQ5 Can the complexity of prepositional phrase understanding be formally assessed using automata-based models?

Key findings

• Semantic complexity can be formally measured using meaning automata, with complexity defined by the number of states and transitions in the automaton.
• The framework enables pre-implementation estimation of semantic complexity, supporting early design evaluation of NLP systems.
• Complex behaviors in natural language understanding, such as those in calendar interfaces, can be decomposed into semantically simple components.
• The study demonstrates that a system with high semantic complexity can be built from components of low individual complexity, suggesting compositional scalability.
• The method provides a formal, quantitative basis for comparing semantic processing systems beyond syntactic or lexical metrics.
• The approach is validated on real-world systems, including prepositional phrase understanding and a natural language calendar interface, showing measurable complexity differences.