[Paper Review] Contextual Sentence Classification: Detecting Sustainability Initiatives in Company Reports

Dan Hirlea, Christopher Bryant+2 more|arXiv (Cornell University)|Oct 7, 2021

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper introduces a novel task for detecting sustainability initiatives in company reports using contextual sentence classification. It proposes a RoBERTa-based model with multi-sentence context and CRF decoding to improve span-level prediction coherence, achieving a 49.16 F1 score on Exact Match with two-sentence context, significantly outperforming binary classification due to better handling of multi-sentence initiatives and label coherence.

ABSTRACT

We introduce the novel task of detecting sustainability initiatives in company reports. Given a full report, the aim is to automatically identify mentions of practical activities that a company has performed in order to tackle specific societal issues. New methods for identifying continuous sentence spans need to be developed for capturing the multi-sentence structure of individual sustainability initiatives. We release a new dataset of company reports in which the text has been manually annotated with sustainability initiatives. We also evaluate different models for initiative detection, introducing a novel aggregation and evaluation methodology. Our proposed architecture uses sequences of consecutive sentences to account for contextual information when making classification decisions at the individual sentence level.

Motivation & Objective

• To address the challenge of detecting sustainability initiatives in company reports, which often span multiple sentences and require semantic understanding beyond isolated sentence analysis.
• To develop a contextual sentence classification framework that leverages surrounding sentences to improve the accuracy and coherence of initiative span detection.
• To release a new, publicly available dataset of 192,978 manually annotated sentences from 45 company reports for benchmarking initiative detection systems.
• To evaluate the impact of context length and label coherence on model performance, particularly for multi-sentence initiatives.
• To explore the scalability and limitations of automated detection, especially in handling subjective or ambiguous cases like singleton initiatives.

Proposed method

• The model uses RoBERTa to jointly encode sequences of multiple consecutive sentences, with a special separator token between sentences to preserve structural boundaries.
• It employs a multi-head classification head for each input sentence, enabling the model to attend to surrounding context during prediction.
• A Conditional Random Field (CRF) is applied to predict IOBES-tagged labels across sentences, enforcing label coherence and improving span boundary detection.
• The model is fine-tuned end-to-end on the annotated dataset, with predictions aggregated into continuous initiative spans for evaluation.
• The architecture is evaluated under varying context window sizes (1 or 2 sentences on each side), showing improved performance with larger context.
• A novel evaluation methodology is introduced, using Min Match and Exact Match metrics to assess the fidelity of predicted initiative spans against gold-standard annotations.

Experimental results

Research questions

• RQ1 How does incorporating multi-sentence context improve the detection of sustainability initiatives compared to single-sentence classification?
• RQ2 To what extent does label coherence, enforced via CRF decoding, enhance the accuracy of initiative span prediction?
• RQ3 What is the optimal context window size for detecting sustainability initiatives, and how does it vary with training data scale?
• RQ4 How do binary versus multi-class IOBES labeling schemes compare in terms of precision, recall, and span-level F1 performance?
• RQ5 Can larger training data improve model generalization and enable better utilization of longer context windows?

Key findings

• The RoBERTa-based model with two-sentence context achieved a 49.16 F1 score on the Exact Match metric, significantly outperforming the 1-sentence context baseline (46.84 F1).
• The use of IOBES labeling with CRF decoding improved prediction coherence and increased the Exact Match F1 by 3.32 points compared to the 1-sentence context baseline.
• The model demonstrated superior performance on multi-sentence initiatives, where context-aware modeling was essential for accurate span detection.
• Adding a larger private training set of 1.5 million sentences improved model performance, enabling better use of longer context windows and increasing F1 scores.
• Binary classification models achieved higher precision but lower recall and were less coherent in predicting multi-sentence initiatives, highlighting the benefit of fine-grained labeling.
• The qualitative analysis revealed that false positives and negatives were most common in singleton initiatives, suggesting inherent subjectivity and annotation fatigue as key challenges.