[Paper Review] Mining Procedures from Technical Support Documents

Abhirut Gupta, Abhay Khosla+2 more|arXiv (Cornell University)|May 24, 2018

Business Process Modeling and AnalysisBusiness, Management and Accounting5 references4 citations

TL;DR

This paper presents a system for mining structured troubleshooting procedures from unstructured technical support documents using NLP and machine learning. It identifies procedures, extracts decision points and associated instruction blocks, and maps conditional logic—enabling applications like intelligent chatbots and RPA by transforming natural language guides into executable workflows with 85% accuracy in condition mapping via text similarity thresholds.

ABSTRACT

Guided troubleshooting is an inherent task in the domain of technical support services. When a customer experiences an issue with the functioning of a technical service or a product, an expert user helps guide the customer through a set of steps comprising a troubleshooting procedure. The objective is to identify the source of the problem through a set of diagnostic steps and observations, and arrive at a resolution. Procedures containing these set of diagnostic steps and observations in response to different problems are common artifacts in the body of technical support documentation. The ability to use machine learning and linguistics to understand and leverage these procedures for applications like intelligent chatbots or robotic process automation, is crucial. Existing research on question answering or intelligent chatbots does not look within procedures or deep-understand them. In this paper, we outline a system for mining procedures from technical support documents. We create models for solving important subproblems like extraction of procedures, identifying decision points within procedures, identifying blocks of instructions corresponding to these decision points and mapping instructions within a decision block. We also release a dataset containing our manual annotations on publicly available support documents, to promote further research on the problem.

Motivation & Objective

• To address the lack of deep understanding of procedures in technical support documents, which are critical for troubleshooting but remain underutilized in NLP systems.
• To extract actionable, structured procedures—including decision points and instruction blocks—from unstructured natural language documentation.
• To enable downstream applications like intelligent chatbots and robotic process automation (RPA) by transforming free-text procedures into machine-understandable workflows.
• To release a manually annotated dataset to support further research in procedure mining from technical documentation.

Proposed method

• Uses a baseline model with features like list structure (e.g., numbered steps) to identify and extract procedures from technical support documents.
• Applies imperative verb detection to improve procedure boundary identification, especially when list features are absent.
• Identifies decision points by detecting conditional sentences (e.g., 'if', 'unless') and uses text similarity (thresholded at 70%) to map instructions to 'True' or 'False' branches.
• Employs rule-based heuristics combined with simple baselines to extract decision blocks, leveraging sentence-level structure and conditional logic.
• Integrates extracted procedures into a knowledge graph for a production chatbot (DESIRE), enabling step-by-step guidance with conditional questions.
• Uses template-based question generation (e.g., 'Do you have X?') to present decision points to users, with response routing via negation detection.

Experimental results

Research questions

• RQ1 How can unstructured technical support documents be automatically mined to extract structured troubleshooting procedures?
• RQ2 What features (e.g., list structure, imperative verbs) most effectively improve procedure boundary detection?
• RQ3 How accurately can conditional logic within procedures be mapped to 'True' or 'False' branches using text similarity?
• RQ4 To what extent can mined procedures enhance the performance of intelligent chatbots or RPA systems?
• RQ5 What challenges remain in mapping instructions to correct execution paths within decision blocks, especially with sparse training data?

Key findings

• About 58% of analyzed IBM technical support documents contain at least one procedure, highlighting the prevalence of this content type.
• Using text similarity above 70% threshold, the system correctly maps 85% of instructions to the correct branch (True or False) in decision blocks.
• Imperative verb detection improves procedure extraction performance, especially when list structure features are unavailable.
• The system successfully integrates with the DESIRE chatbot, enabling guided troubleshooting through step-by-step, condition-aware dialog flows.
• Despite promising results, mapping instructions within decision blocks remains challenging due to limited annotated data, indicating a need for more extensive datasets.
• The authors plan to release the annotated dataset to support future research in procedure mining from technical documentation.