[Paper Review] Active Learning for Automated Visual Inspection of Manufactured Products

Elena Trajkova, Jože M. Rožanec+3 more|arXiv (Cornell University)|Sep 6, 2021

Machine Learning and Algorithms10 references4 citations

TL;DR

This paper proposes an active learning framework for automated visual inspection in manufacturing, comparing pool-based, stream-based, and query-by-committee strategies with five machine learning models on real-world shaver defect images. The MLP model achieved the highest AUC-ROC (0.9840), and query-by-committee outperformed other active learning methods, reducing labeling effort without sacrificing performance.

ABSTRACT

Quality control is a key activity performed by manufacturing enterprises to ensure products meet quality standards and avoid potential damage to the brand's reputation. The decreased cost of sensors and connectivity enabled an increasing digitalization of manufacturing. In addition, artificial intelligence enables higher degrees of automation, reducing overall costs and time required for defect inspection. In this research, we compare three active learning approaches and five machine learning algorithms applied to visual defect inspection with real-world data provided by Philips Consumer Lifestyle BV. Our results show that active learning reduces the data labeling effort without detriment to the models' performance.

Motivation & Objective

• To evaluate active learning strategies in reducing labeling effort for industrial visual defect detection.
• To compare the performance of five machine learning models—Gaussian Naïve Bayes, CART, Linear SVM, MLP, and kNN—on a real-world multiclass defect classification task.
• To assess whether active learning improves model performance with fewer labeled examples, particularly in imbalanced industrial datasets.
• To determine if performance gains are statistically significant over time during active learning cycles.
• To identify the most effective combination of active learning strategy and machine learning model for automated visual inspection in manufacturing.

Proposed method

• Applied three active learning strategies: pool-based sampling, stream-based sampling with uncertainty sampling, and query-by-committee.
• Trained five machine learning models—Gaussian Naïve Bayes, CART, Linear SVM, MLP, and k-Nearest Neighbors—on a real-world dataset of 1,000 shaver images with three defect classes: good, double printing, and interrupted printing.
• Used 10-fold cross-validation to evaluate model performance, measuring AUC-ROC at a 0.5 threshold for prediction scores.
• Selected the most informative samples using uncertainty-based acquisition functions in pool- and stream-based settings, and committee disagreement in query-by-committee.
• Applied the Wilcoxon signed-rank test to assess statistical significance of performance differences between active learning scenarios and across model performance over time.
• Evaluated model improvement between first and last quartiles of test folds to assess learning progression during active learning cycles.

Experimental results

Research questions

• RQ1 Which active learning strategy—pool-based, stream-based, or query-by-committee—yields the highest defect detection performance with minimal labeling effort?
• RQ2 How do different machine learning models (Naïve Bayes, CART, SVM, MLP, kNN) perform across the three active learning strategies in a real industrial defect detection scenario?
• RQ3 Is there a statistically significant improvement in model performance over time during active learning cycles, particularly between early and late stages of labeling?
• RQ4 Does the query-by-committee strategy consistently outperform other active learning methods across different models and data splits?
• RQ5 Can active learning effectively reduce labeling costs while maintaining high AUC-ROC performance in imbalanced, real-world manufacturing datasets?

Key findings

• The MLP model achieved the highest AUC-ROC (0.9840) across all active learning scenarios, outperforming all other models in most cross-validation folds.
• Query-by-committee active learning yielded the second-highest performance, with AUC-ROC values consistently above 0.97, and showed statistically significant improvement over all models except MLP.
• No significant performance difference was found between pool-based and stream-based active learning strategies, as indicated by p-values > 0.05 in most comparisons.
• The SVM model ranked third overall, with AUC-ROC values around 0.972, and showed no statistically significant difference across active learning strategies (p > 0.05).
• The CART model performed the worst, with AUC-ROC values below 0.78, and showed no significant improvement over time in any active learning scenario.
• No statistically significant improvement was observed between the first and last quartiles of test folds across any active learning strategy, indicating limited learning progression during labeling cycles.