[Paper Review] A Survey of Deep Learning for Scientific Discovery

Maithra Raghu, E. Schmidt|arXiv (Cornell University)|Mar 26, 2020

Machine Learning and Data Classification249 references58 citations

TL;DR

This survey reviews how deep learning models across data modalities can aid scientific discovery, emphasizing data efficiency, interpretability, and practical implementation resources.

ABSTRACT

Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. At the same time, the amount of data collected in a wide array of scientific domains is dramatically increasing in both size and complexity. Taken together, this suggests many exciting opportunities for deep learning applications in scientific settings. But a significant challenge to this is simply knowing where to start. The sheer breadth and diversity of different deep learning techniques makes it difficult to determine what scientific problems might be most amenable to these methods, or which specific combination of methods might offer the most promising first approach. In this survey, we focus on addressing this central issue, providing an overview of many widely used deep learning models, spanning visual, sequential and graph structured data, associated tasks and different training methods, along with techniques to use deep learning with less data and better interpret these complex models --- two central considerations for many scientific use cases. We also include overviews of the full design process, implementation tips, and links to a plethora of tutorials, research summaries and open-sourced deep learning pipelines and pretrained models, developed by the community. We hope that this survey will help accelerate the use of deep learning across different scientific domains.

Motivation & Objective

• Provide a broad, accessible overview of deep learning concepts applicable to scientific problems.
• Highlight data-efficient training methods (self-supervision, semi-supervised learning) and interpretability techniques relevant to science.
• Outline an end-to-end deep learning workflow in scientific contexts, including data, learning, and validation stages.
• Offer implementation guidance, tutorials, and open-source resources to accelerate adoption in science.

Proposed method

• Survey multiple deep learning models (CNNs, GNNs, RNNs, Transformers) and their typical scientific tasks (classification, segmentation, registration).
• Discuss training methods including supervised, self-supervised, semi-supervised, and transfer learning.
• Present templates for applying DL in science (prediction, understanding, complex transformations).
• Describe data efficiency strategies (augmentation, denoising) and interpretability/representation analysis techniques.
• Provide implementation tips and enumerate community resources, tutorials, and pretrained models.

Experimental results

Research questions

• RQ1 Which deep learning models and tasks are most suitable for specific scientific problems?
• RQ2 How can one achieve data-efficient training and reliable interpretability in scientific DL applications?
• RQ3 What practical resources (code, tutorials, pretrained models) best accelerate adoption in scientific domains?
• RQ4 What is the end-to-end workflow for designing, validating, and deploying DL systems in science?
• RQ5 How do alternative ML methods compare to DL in various scientific contexts?

Key findings

• Provides a structured overview of models, tasks, and training methods across visual, sequential, and graph data relevant to science.
• Highlights data-efficient approaches (self-supervision, semi-supervised learning, data augmentation) and interpretability techniques critical for scientific insight.
• Outlines an end-to-end DL design process with iterative data, learning, and validation loops.
• Offers a curated set of tutorials, open-source code, pretrained models, and community resources to accelerate adoption.
• Notes that DL is powerful for complex transformations and predictions but may not always be the best first tool; recommends considering alternative ML methods when appropriate.