[Paper Review] Accelerating science with human-aware artificial intelligence

Jamshid Sourati, James Evans|arXiv (Cornell University)|Jun 2, 2023

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes human-aware artificial intelligence that models human expertise distributions to predict future scientific discoveries, significantly improving AI performance by simulating cognitively accessible inferences for experts. By incorporating human cognitive constraints and expertise, the method boosts prediction accuracy by up to 400% in sparse literature settings and generates 'alien' hypotheses unlikely to emerge through conventional research paths.

ABSTRACT

Artificial intelligence (AI) models trained on published scientific findings have been used to invent valuable materials and targeted therapies, but they typically ignore the human scientists who continually alter the landscape of discovery. Here we show that incorporating the distribution of human expertise by training unsupervised models on simulated inferences cognitively accessible to experts dramatically improves (up to 400%) AI prediction of future discoveries beyond those focused on research content alone, especially when relevant literature is sparse. These models succeed by predicting human predictions and the scientists who will make them. By tuning human-aware AI to avoid the crowd, we can generate scientifically promising "alien" hypotheses unlikely to be imagined or pursued without intervention until the distant future, which hold promise to punctuate scientific advance beyond questions currently pursued. Accelerating human discovery or probing its blind spots, human-aware AI enables us to move toward and beyond the contemporary scientific frontier.

Motivation & Objective

• To address the limitation of existing AI models in scientific discovery that ignore human cognitive and expertise constraints.
• To improve AI prediction of future scientific discoveries by modeling the distribution of human expertise and cognitively accessible inferences.
• To generate scientifically promising hypotheses that are unlikely to be conceived or pursued by current research communities without intervention.
• To enable AI to probe scientific blind spots and accelerate discovery beyond the current frontier.

Proposed method

• Train unsupervised models on simulated inferences that are cognitively accessible to human experts, rather than relying solely on research content.
• Incorporate the distribution of human expertise into the representation learning process to reflect who is likely to make which inferences.
• Use cognitive plausibility as a proxy for scientific novelty by modeling what experts are likely to predict.
• Tune the AI to avoid common human prediction patterns, generating 'alien' hypotheses that diverge from mainstream research trajectories.
• Leverage sparse literature scenarios to test the model’s ability to outperform content-only models.
• Validate predictions against known scientific discoveries and assess novelty through human expert evaluation.

Experimental results

Research questions

• RQ1 How does incorporating human expertise distributions improve AI's ability to predict future scientific discoveries compared to content-only models?
• RQ2 To what extent can human-aware AI generate hypotheses that are scientifically promising but unlikely to be pursued by current research communities?
• RQ3 Can models trained on cognitively accessible inferences outperform standard NLP models in low-data or sparse literature settings?
• RQ4 How does tuning the AI to avoid the 'crowd' produce hypotheses that could punctuate scientific progress?
• RQ5 What is the quantitative impact of human-aware AI on prediction accuracy in scientific discovery tasks?

Key findings

• Incorporating human expertise distributions into AI models improved prediction of future discoveries by up to 400% compared to content-only models, especially in sparse literature settings.
• The human-aware AI model successfully predicted discoveries by simulating which scientists were likely to make which inferences, aligning with real-world expert behavior.
• By tuning the model to avoid common human prediction patterns, it generated 'alien' hypotheses—scientifically plausible but overlooked—suggesting new paths for discovery.
• The model demonstrated superior performance in low-data regimes, where traditional AI models struggle due to lack of training signals.
• The approach enables identification of scientific blind spots and accelerates discovery by generating hypotheses that would otherwise remain unexplored for decades.