[Paper Review] What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text

Peter E. Clark, Bhavana Dalvi+1 more|arXiv (Cornell University)|Apr 15, 2018

Topic Modeling25 references13 citations

TL;DR

This paper presents ProComp, a system that leverages a VerbNet-derived Semantic Lexicon to model the preconditions and effects of actions in procedural texts, enabling inference about implicit world state changes. By simulating state transitions and integrating commonsense persistence reasoning, ProComp significantly outperforms surface-cue-based reading comprehension baselines on question-answering tasks involving process descriptions.

ABSTRACT

Our goal is to answer questions about paragraphs describing processes (e.g., photosynthesis). Texts of this genre are challenging because the effects of actions are often implicit (unstated), requiring background knowledge and inference to reason about the changing world states. To supply this knowledge, we leverage VerbNet to build a rulebase (called the Semantic Lexicon) of the preconditions and effects of actions, and use it along with commonsense knowledge of persistence to answer questions about change. Our evaluation shows that our system, ProComp, significantly outperforms two strong reading comprehension (RC) baselines. Our contributions are two-fold: the Semantic Lexicon rulebase itself, and a demonstration of how a simulation-based approach to machine reading can outperform RC methods that rely on surface cues alone. Since this work was performed, we have developed neural systems that outperform ProComp, described elsewhere (Dalvi et al., NAACL'18). However, the Semantic Lexicon remains a novel and potentially useful resource, and its integration with neural systems remains a currently unexplored opportunity for further improvements in machine reading about processes.

Motivation & Objective

• To address the challenge of answering questions about implicit changes in world states described in procedural texts, where effects are not explicitly stated.
• To develop a rule-based system that models how actions alter the world state using linguistic event patterns.
• To demonstrate that simulation-based reasoning with semantic knowledge outperforms reading comprehension models relying solely on surface cues.
• To create a reusable, knowledge-rich resource—the Semantic Lexicon—for reasoning about action effects in process descriptions.

Proposed method

• The Semantic Lexicon is constructed by extending VerbNet with rules that encode preconditions and effects of actions as logical implications (e.g., 'enter' implies 'is-at' after the event).
• ProComp parses procedural texts into a process graph, tracking entities and events across time steps.
• It applies the Semantic Lexicon rules to infer new world state literals (e.g., 'CO2 is at the leaf' after 'enters') at each event step.
• Persistence reasoning is applied to maintain state across time steps, assuming unchanged facts unless modified by an action.
• The system uses a participant grid to represent the state of entities before and after each event, enabling multi-step inference.
• It answers questions by querying the final state of the simulation, combining logical inference with coreference resolution.

Experimental results

Research questions

• RQ1 Can a rule-based system that models action effects using a semantic lexicon outperform neural reading comprehension models on questions requiring inference about implicit changes in procedural texts?
• RQ2 To what extent does integrating world state simulation with language understanding improve reasoning over surface-level cues?
• RQ3 How effective is a VerbNet-derived rulebase in capturing the preconditions and effects of actions in real-world process descriptions?
• RQ4 What types of errors arise when relying on symbolic reasoning for complex world knowledge, and how do they compare to neural models' reliance on patterns?

Key findings

• ProComp achieved an F1 score of 74.8 on all questions and 67.0 on single-answer questions, significantly outperforming two strong RC baselines.
• Ablation studies showed that removing VerbNet extensions reduced F1 by 5.9 points, confirming the importance of the Semantic Lexicon’s enhancements.
• The system demonstrated superior performance on multi-event inference questions, such as tracking the origin of ingots after a series of transport and transformation events.
• NLP pipeline errors accounted for ~20% of failures, indicating that robust event extraction remains a bottleneck.
• Verb semantics omissions (e.g., missing location change for 'evaporate') contributed to ~30% of errors, highlighting gaps in the lexicon.
• Complex reasoning beyond the system’s scope—such as inferring hailstone formation or temperature changes in freezers—accounted for ~40% of errors, revealing limits of symbolic rule-based modeling.