[Paper Review] Rationale in Development Chat Messages: An Exploratory Study

Rana Alkadhi, Teodora Lata+2 more|arXiv (Cornell University)|Apr 27, 2017

Software Engineering Research33 references12 citations

TL;DR

This study investigates chat messages from software development teams as a rich, untapped source of rationale—justifications, decisions, and design trade-offs—using content analysis and machine learning. It demonstrates that machine learning can effectively detect rationale in chat messages with up to 0.88 recall and filter out non-relevant messages with 0.98 recall, enabling scalable extraction of development knowledge from informal team communication.

ABSTRACT

Chat messages of development teams play an increasingly significant role in software development, having replaced emails in some cases. Chat messages contain information about discussed issues, considered alternatives and argumentation leading to the decisions made during software development. These elements, defined as rationale, are invaluable during software evolution for documenting and reusing development knowledge. Rationale is also essential for coping with changes and for effective maintenance of the software system. However, exploiting the rationale hidden in the chat messages is challenging due to the high volume of unstructured messages covering a wide range of topics. This work presents the results of an exploratory study examining the frequency of rationale in chat messages, the completeness of the available rationale and the potential of automatic techniques for rationale extraction. For this purpose, we apply content analysis and machine learning techniques on more than 8,700 chat messages from three software development projects. Our results show that chat messages are a rich source of rationale and that machine learning is a promising technique for detecting rationale and identifying different rationale elements.

Motivation & Objective

• To investigate the frequency and completeness of rationale in informal chat messages used by software development teams.
• To evaluate the feasibility of using machine learning techniques for automatically detecting and classifying rationale elements in chat messages.
• To assess the potential of automated methods for extracting tacit development knowledge from unstructured, real-time team communication.
• To provide empirical evidence that chat messages are a viable alternative to formal rationale documentation, which is often underused due to overhead.

Proposed method

• Conducted content analysis on 8,702 chat messages from three software development projects to manually label rationale elements such as issues, alternatives, pro/con-arguments, and decisions.
• Used the manually labeled dataset to train and evaluate multiple machine learning models for rationale detection and classification.
• Applied supervised learning techniques to classify messages at two granularity levels: (1) presence of rationale, and (2) identification of specific rationale elements.
• Employed linguistic and structural features from chat messages as input for the machine learning models, focusing on informal, short-form communication patterns.
• Evaluated model performance using standard metrics including precision, recall, and F1-score to assess detection and classification accuracy.

Experimental results

Research questions

• RQ1 How frequently do rationale elements appear in software development chat messages?
• RQ2 How complete is the rationale captured in these messages, and what types of rationale elements are most prevalent?
• RQ3 Can machine learning techniques effectively detect messages containing rationale in informal chat communication?
• RQ4 Can machine learning models accurately classify detected rationale into specific elements such as issues, alternatives, arguments, and decisions?

Key findings

• Chat messages contain a significant amount of rationale, with 44.7% of messages containing at least one rationale element, indicating they are a rich source of development knowledge.
• Machine learning models achieved a recall of up to 0.88 in detecting messages containing rationale, demonstrating strong potential for automated extraction.
• Models could filter out messages without rationale with a recall of up to 0.98, significantly reducing noise in large message corpora.
• Fine-grained classification of rationale elements (e.g., issue, alternative, pro/con-argument, decision) is feasible, with promising performance across different models.
• The results suggest that informal chat communication is a viable and underutilized medium for capturing and reusing software development rationale.