[Paper Review] Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning

Sara Abdali, Anjali Parikh+2 more|arXiv (Cornell University)|Dec 11, 2023

Software Engineering Research4 citations

TL;DR

This paper proposes a self-consistent, in-context learning framework using large language models (LLMs) to extract causal variables and event sequences from user feedback in the Windows Feedback Hub. By leveraging LLM reasoning and a modified self-consistency approach, the method identifies treatment, outcome, confounders, and causal chains with 0% hallucination, achieving 81% known issues and 19% novel bug discovery while scoring feedback for actionability.

ABSTRACT

Microsoft Windows Feedback Hub is designed to receive customer feedback on a wide variety of subjects including critical topics such as power and battery. Feedback is one of the most effective ways to have a grasp of users' experience with Windows and its ecosystem. However, the sheer volume of feedback received by Feedback Hub makes it immensely challenging to diagnose the actual cause of reported issues. To better understand and triage issues, we leverage Double Machine Learning (DML) to associate users' feedback with telemetry signals. One of the main challenges we face in the DML pipeline is the necessity of domain knowledge for model design (e.g., causal graph), which sometimes is either not available or hard to obtain. In this work, we take advantage of reasoning capabilities in Large Language Models (LLMs) to generate a prior model that which to some extent compensates for the lack of domain knowledge and could be used as a heuristic for measuring feedback informativeness. Our LLM-based approach is able to extract previously known issues, uncover new bugs, and identify sequences of events that lead to a bug, while minimizing out-of-domain outputs.

Motivation & Objective

• Address the challenge of diagnosing root causes in high-volume user feedback for Windows OS issues, particularly for complex, multi-factor problems like power and battery drain.
• Overcome the reliance on expert-provided causal graphs and domain knowledge in Double Machine Learning (DML) pipelines by using LLMs to generate heuristic priors.
• Minimize LLM hallucination in causal variable extraction through a modified self-consistency framework using ensemble prompting.
• Develop causal heuristics to score feedback informativeness based on treatment-outcome relationships and sequence complexity.
• Enable engineers to triage feedback more efficiently by extracting actionable causal summaries and identifying previously unknown causal chains.

Proposed method

• Employ in-context learning (ICL) with few-shot and zero-shot chain-of-thought prompting to guide LLMs in identifying causal variables: treatment, outcome, confounders, and instrumental variables.
• Implement a modified self-consistency framework that ensembles multiple prompt variations and uses a greedy selection strategy to improve consistency and reduce hallucination.
• Extract sequences of events from feedback using LLMs to reconstruct causal narratives leading to reported bugs.
• Design two causal heuristics: (1) sum of unique causal variables (treatment, outcome, confounders), and (2) cumulative length of extracted event chains, to score feedback actionability.
• Validate extracted causal variables and chains against known engineering triage data to ensure in-domain accuracy and minimize out-of-domain outputs.
• Use Double Machine Learning (DML) as the downstream causal inference pipeline, with LLM-generated priors informing model structure and confounder selection.

Experimental results

Research questions

• RQ1 Can LLMs with in-context learning reliably extract self-consistent causal variables (treatment, outcome, confounders) from unstructured user feedback without expert-provided causal graphs?
• RQ2 To what extent does a modified self-consistency prompting strategy reduce hallucination in LLM-based causal reasoning compared to standard prompting?
• RQ3 Can LLMs discover previously unknown causal chains in user feedback that are relevant to diagnosing OS-level bugs such as battery drain or sleep failures?
• RQ4 How effective are the proposed causal heuristics in predicting feedback actionability compared to human triage standards?
• RQ5 What is the proportion of extracted insights that are either pre-classified known issues or novel, engineer-verified bugs?

Key findings

• The modified self-consistency framework reduced hallucination to 0% in causal variable extraction, ensuring all outputs were in-domain.
• 81% of extracted causal variables corresponded to previously known issues, while 19% revealed new, previously undocumented bugs.
• 44% of extracted causal event chains were previously unknown to engineers, indicating the method's ability to uncover novel diagnostic pathways.
• Feedback scored using heuristic 2 (cumulative sequence length) showed higher actionability, especially when multiple chains were extracted from a single feedback.
• The method successfully generated a prior model for causal inference that supported downstream DML pipelines, even in the absence of explicit domain knowledge.
• All extracted causal insights were relevant to the target topic (e.g., 'Modern Standby'), with no out-of-domain responses observed.