[Paper Review] DeepFreak: Learning Crystallography Diffraction Patterns with Automated Machine Learning

Artur Souza, Leonardo B. Oliveira+6 more|arXiv (Cornell University)|Apr 26, 2019

X-ray Diffraction in Crystallography33 references4 citations

TL;DR

This paper introduces DiffraNet, a novel 25,457-image dataset combining real and synthetic X-ray diffraction patterns for serial crystallography, and proposes DeepFreak, a custom CNN architecture fine-tuned via AutoML. The best model achieves 98.5% accuracy on synthetic images and 94.51% on real images, demonstrating strong generalization from simulation to real-world data.

ABSTRACT

Serial crystallography is the field of science that studies the structure and properties of crystals via diffraction patterns. In this paper, we introduce a new serial crystallography dataset comprised of real and synthetic images; the synthetic images are generated through the use of a simulator that is both scalable and accurate. The resulting dataset is called DiffraNet, and it is composed of 25,457 512x512 grayscale labeled images. We explore several computer vision approaches for classification on DiffraNet such as standard feature extraction algorithms associated with Random Forests and Support Vector Machines but also an end-to-end CNN topology dubbed DeepFreak tailored to work on this new dataset. All implementations are publicly available and have been fine-tuned using off-the-shelf AutoML optimization tools for a fair comparison. Our best model achieves 98.5% accuracy on synthetic images and 94.51% accuracy on real images. We believe that the DiffraNet dataset and its classification methods will have in the long term a positive impact in accelerating discoveries in many disciplines, including chemistry, geology, biology, materials science, metallurgy, and physics.

Motivation & Objective

• Address the critical bottleneck in serial crystallography where manual classification of diffraction images is slow and error-prone.
• Develop a scalable, accurate method for generating labeled diffraction images to overcome the limitations of manual annotation.
• Create a publicly available, high-quality dataset (DiffraNet) that combines real and synthetic diffraction patterns for training and benchmarking.
• Evaluate multiple machine learning approaches—traditional ML with feature extractors and deep learning with a custom CNN—on the new dataset.
• Demonstrate that models trained exclusively on synthetic data can generalize effectively to real diffraction images, enabling real-time feedback in crystallography experiments.

Proposed method

• Develop a physics-based simulator that generates accurate, scalable synthetic diffraction images by modeling X-ray beam properties, crystal structure, and experimental environment.
• Use the simulator to generate 25,457 labeled 512×512 grayscale images across five classes: two indicating no diffraction (undesired outcomes), and three indicating varying degrees of diffraction (desired outcomes).
• Collect and label 457 real diffraction images from actual serial crystallography experiments, categorized into two classes: with or without diffraction patterns.
• Construct the DiffraNet dataset by combining synthetic and real images, ensuring diverse and realistic representation of diffraction patterns.
• Implement three classification methods: (1) Random Forest with GLCM and LBP features, (2) SVM with same features, and (3) an end-to-end CNN named DeepFreak based on ResNet-50 architecture.
• Fine-tune all models using AutoML tools (Hyperopt and BOHB) for hyperparameter optimization, with separate optimization loops for synthetic and real data to improve generalization.

Experimental results

Research questions

• RQ1 Can a physics-based simulator generate synthetic diffraction images with sufficient accuracy and scalability to support machine learning training in crystallography?
• RQ2 To what extent can models trained exclusively on synthetic diffraction images generalize to real-world diffraction patterns in serial crystallography?
• RQ3 How do traditional machine learning models (e.g., RF, SVM with handcrafted features) compare to deep learning models (e.g., CNNs) in classifying diffraction patterns from real and synthetic data?
• RQ4 What is the impact of the 'reality gap'—the performance drop between synthetic and real data—on different classification architectures?
• RQ5 Can AutoML optimization improve model generalization to real diffraction images without requiring real data in the training set?

Key findings

• The DeepFreak CNN model achieved the highest accuracy on both synthetic and real diffraction images, reaching 98.5% on synthetic data and 94.51% on real data after AutoML fine-tuning.
• The reality gap significantly affected model performance, with the Random Forest model suffering a 54.56% accuracy drop on real data, while DeepFreak was the least affected, losing only 22.45% accuracy.
• After re-optimizing models using the real dataset as a validation set (without adding real data to training), DeepFreak achieved 94.51% accuracy on the preprocessed real test set, demonstrating strong generalization from synthetic data.
• The model trained on synthetic data alone achieved 99.83% accuracy in distinguishing between images with and without diffraction patterns, indicating high sensitivity to the presence of diffraction.
• The SVM and Random Forest models showed poor generalization to real data, with accuracy drops exceeding 50% in some cases, highlighting the limitations of traditional ML approaches on this domain.
• The results confirm that high-quality synthetic data generated via physics-based simulation can effectively train models for real-world crystallography applications, reducing reliance on costly manual labeling.