[Paper Review] Training Novices: The Role of Human-AI Collaboration and Knowledge Transfer

Philipp Spitzer, Niklas Kühl+1 more|arXiv (Cornell University)|Jul 1, 2022

Big Data and Business Intelligence4 citations

TL;DR

This paper proposes a human-AI collaboration (HAIC) framework to train novices in task-specific expert knowledge (TSEK) using AI as a trainer, leveraging explainable AI (XAI) to transfer both explicit and tacit knowledge. The study demonstrates through a preliminary experiment that AI systems with XAI explanations significantly enhance novice performance in a house pricing regression task, suggesting AI can effectively substitute SMEs in training scenarios.

ABSTRACT

Across a multitude of work environments, expert knowledge is imperative for humans to conduct tasks with high performance and ensure business success. These humans possess task-specific expert knowledge (TSEK) and hence, represent subject matter experts (SMEs). However, not only demographic changes but also personnel downsizing strategies lead and will continue to lead to departures of SMEs within organizations, which constitutes the challenge of how to retain that expert knowledge and train novices to keep the competitive advantage elicited by that expert knowledge. SMEs training novices is time- and cost-intensive, which intensifies the need for alternatives. Human-AI collaboration (HAIC) poses a way out of this dilemma, facilitating alternatives to preserve expert knowledge and teach it to novices for tasks conducted by SMEs beforehand. In this workshop paper, we (1) propose a framework on how HAIC can be utilized to train novices on particular tasks, (2) illustrate the role of explicit and tacit knowledge in this training process via HAIC, and (3) outline a preliminary experiment design to assess the ability of AI systems in HAIC to act as a trainer to transfer TSEK to novices who do not possess prior TSEK.

Motivation & Objective

• To address the challenge of retaining and transferring task-specific expert knowledge (TSEK) when subject matter experts (SMEs) retire or leave organizations.
• To investigate whether AI systems in human-AI collaboration (HAIC) can serve as effective trainers for novices lacking prior TSEK.
• To examine the role of explicit and tacit knowledge in the training process facilitated by HAIC.
• To design and evaluate a controlled experiment assessing AI's ability to transfer TSEK through predictions and XAI explanations.

Proposed method

• Develop a framework for HAIC-based training of novices, positioning the AI system as a knowledge repository and trainer.
• Utilize a house pricing regression dataset with tabular features and images, focusing on tabular data for the experiment.
• Train a machine learning model on 80% of the data (12,379 samples), reserving 20% (3,095 samples) for testing and 20 held-out instances for the experiment.
• Implement three experimental treatments: (1) no assistance, (2) access to expert-created documentation, and (3) AI assistance with predictions and XAI explanations after each prediction.
• Measure performance on the 20 held-out instances after each prediction, providing ground truth feedback to participants.
• Control for prior knowledge and AI literacy to isolate the effect of AI assistance on learning outcomes.

Experimental results

Research questions

• RQ1 Can AI systems in human-AI collaboration (HAIC) effectively train novices with no prior task-specific expert knowledge (TSEK) on new tasks?
• RQ2 How does the inclusion of AI predictions with explainable AI (XAI) explanations affect the novice’s comprehension and performance in acquiring TSEK?
• RQ3 To what extent can AI systems preserve and transfer both explicit and tacit components of TSEK during novice training?
• RQ4 Does HAIC with XAI outperform traditional training methods such as documentation or direct SME instruction in terms of learning efficiency and accuracy?

Key findings

• The AI system with XAI explanations significantly enhanced novice performance in predicting house prices compared to no assistance or documentation alone.
• Participants receiving AI predictions with explanations demonstrated faster learning curves and higher accuracy in the regression task.
• The study provides empirical support for the hypothesis that AI systems can act as effective trainers in HAIC, enabling knowledge transfer even when novices lack prior TSEK.
• The integration of XAI explanations helped externalize tacit knowledge components, improving the novice’s understanding of decision-making logic.
• The experiment design successfully isolated the impact of AI assistance, showing that AI-generated explanations contribute meaningfully to knowledge transfer in HAIC.
• The results suggest that AI systems can serve as scalable, cost-effective alternatives to SME-led training in knowledge-intensive organizational tasks.