[Paper Review] NUBIA: NeUral Based Interchangeability Assessor for Text Generation

Hassan Kané, Muhammed Yusuf Kocyigit+3 more|arXiv (Cornell University)|Apr 30, 2020

Natural Language Processing Techniques28 references36 citations

TL;DR

NUBIA proposes a neural feature-based learned metric for text generation that combines transformer-derived features with an aggregator and calibrator to predict human judgments, achieving strong correlations on machine translation and image captioning benchmarks.

ABSTRACT

We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA which outperforms metrics currently used to evaluate machine translation, summaries and slightly exceeds/matches state of the art metrics on correlation with human judgement on the WMT segment-level Direct Assessment task, sentence-level ranking and image captioning evaluation. The model implemented is modular, explainable and set to continuously improve over time.

Motivation & Objective

• Motivation to move beyond BLEU/ROUGE toward learned, domain-unified evaluation metrics for text generation.
• Develop a modular, explainable metric that can leverage advances in neural representations.
• Demonstrate strong correlation with human judgments across machine translation and image captioning tasks.
• Show that the metric can improve over time with better features and aggregators.

Proposed method

• Three-module architecture: neural feature extractor, aggregator, and calibrator.
• Neural features include semantic similarity (RoBERTa fine-tuned on STS-B), logical inference (MNLI fine-tuning), and sentence legibility (GPT-2 perplexity).
• Aggregator is a regression model (linear or neural network) mapping features to a quality score; neural aggregator variants use 6 or 8 input features with 10 hidden layers and tanh activations.
• Calibrator caps outputs to [0,1] and normalizes scores by self-comparison to ensure self-consistency.
• Training uses WMT data (2015–2017) and evaluates on WMT 2017–2019 and Flickr 8K for image captioning.
• Experiments compare against BLEU, ROUGE, BERTScore, YiSi, RUSE, etc., using Pearson or Kendall’s Tau as appropriate.
• Ablation studies identify the semantic similarity feature as most crucial, with additional gains from MNLI and GPT-2 features.

Experimental results

Research questions

• RQ1 Can a neural feature-based architecture unify evaluation across MT and image captioning tasks?
• RQ2 Do transformer-derived features correlate more strongly with human judgments than traditional n-gram based metrics?
• RQ3 What is the contribution of individual features (semantic similarity, logical inference, sentence intelligibility) to predicting human scores?
• RQ4 Can the proposed metric achieve state-of-the-art correlations with human judgments on standard benchmarks?
• RQ5 Is the approach robust across languages and datasets, and how does it handle calibration and self-consistency?

Key findings

• NUBIA-8DIM-NN and NUBIA-8DIM-LREG achieve very high absolute correlations with human judgments on MT17–MT19 benchmarks, often outperforming BLEU, ROUGE, and several neural baselines.
• On Flickr 8K image captioning, NUBIA variants surpass traditional metrics like BLEU, ROUGE-L, METEOR, CIDEr, and SPICE in Kendall’s Tau correlations with human scores, reaching top performance in the reported setup.
• Ablation shows RoBERTa semantic similarity as the most crucial feature; adding MNLI and GPT-2-based features yields further improvements.
• The method yields state-of-the-art results for segment-level ranking on MT18 and competitive results on MT19, demonstrating strong cross-task applicability.
• The architecture is modular and explainable, with potential for continuous improvement as better semantic similarity, coherence, and language modeling features emerge.