[Paper Review] Fine-Grained Sentence Functions for Short-Text Conversation

Wei Bi, Jun Gao+2 more|arXiv (Cornell University)|Jul 24, 2019

Topic Modeling20 references4 citations

TL;DR

This paper introduces STC-Sefun, a large-scale short-text conversation dataset with fine-grained sentence function annotations (20 types across two levels), enabling improved response generation in dialogue systems. By training classifiers to predict sentence functions and integrating them into retrieval and generative models, the authors demonstrate significant gains in response relevance, informativeness, and fluency, with re-ranked IR models outperforming both baselines and generative models on key metrics.

ABSTRACT

Sentence function is an important linguistic feature referring to a user's purpose in uttering a specific sentence. The use of sentence function has shown promising results to improve the performance of conversation models. However, there is no large conversation dataset annotated with sentence functions. In this work, we collect a new Short-Text Conversation dataset with manually annotated SEntence FUNctions (STC-Sefun). Classification models are trained on this dataset to (i) recognize the sentence function of new data in a large corpus of short-text conversations; (ii) estimate a proper sentence function of the response given a test query. We later train conversation models conditioned on the sentence functions, including information retrieval-based and neural generative models. Experimental results demonstrate that the use of sentence functions can help improve the quality of the returned responses.

Motivation & Objective

• To address the lack of large-scale, manually annotated short-text conversation datasets with sentence function labels.
• To develop reliable sentence function classification models for both identifying sentence functions in utterances and predicting appropriate response sentence functions for given queries.
• To investigate how incorporating sentence functions improves response quality in retrieval-based and neural generative dialogue models.
• To evaluate the impact of sentence function modeling on response relevance, informativeness, fluency, and accuracy across multiple dialogue system architectures.

Proposed method

• Annotate 10,000 query-response pairs from short-text conversations with two-level sentence function labels: 4 coarse-level (Interrogative, Declarative, Imperative, Exclamatory) and 20 fine-grained subtypes (e.g., Wh-style Interrogative, Yes-no Interrogative).
• Train two separate sentence function classification models: one for classifying sentence functions in unlabeled data, and another for predicting the target response sentence function given a query.
• Construct a large automatically annotated corpus using the trained classifiers to enable training and evaluation of dialogue models at scale.
• Enhance retrieval-based models by re-ranking candidate responses using a hybrid score that combines Jaccard similarity and compatibility with the predicted target sentence function.
• Extend sequence-to-sequence and CVAE-based generative models by conditioning on sentence function embeddings, including a type discriminator in SeFun-CVAE to better disentangle sentence function information.
• Evaluate models using standard metrics: fluency, relevance, informativeness, and accuracy of predicted sentence functions.

Experimental results

Research questions

• RQ1 Can fine-grained sentence function annotations improve the quality of responses in short-text conversation systems?
• RQ2 How accurately can sentence function be predicted for both input queries and potential responses in a large-scale short-text conversation corpus?
• RQ3 To what extent does conditioning retrieval-based and generative dialogue models on predicted sentence functions enhance response relevance and informativeness?
• RQ4 Which model architecture—retrieval-based or generative—benefits more from incorporating sentence function information in terms of response quality and accuracy?

Key findings

• The re-ranked IR model achieved 69.6% fluency, 74.4% relevance, 77.2% informativeness, and 50.5% accuracy on the target sentence function prediction, significantly outperforming the IR baseline (63.4%, 68.4%, 61.5%, 34.3%).
• For generative models, C-Seq2seq achieved the highest overall performance, with 58.9% relevance and 37.2% accuracy on level-2 sentence functions, outperforming the Seq2seq baseline and other variants.
• The SeFun-CVAE model, which includes a type discriminator, achieved higher sentence function accuracy (36.7%) than the baseline KgCVAE (33.7%), demonstrating the effectiveness of explicit disentanglement of sentence function in latent space.
• Despite lower fluency and relevance scores, the best generative models (SeFun-CVAE and C-Seq2seq) outperformed the IR baseline on sentence function accuracy, indicating that conditioning on sentence function improves control over response type.
• The re-ranked IR model consistently outperformed all generative models on all metrics, suggesting that sentence function integration is more effective in retrieval-based frameworks for short-text conversations.
• Case studies confirm that re-ranked IR models produce more coherent and functionally appropriate responses, even when term overlap with the query is low, due to better alignment with the target sentence function.