[Paper Review] Comparative Study of Instance Based Learning and Back Propagation for Classification Problems

Nadia Kanwal, Erkan Bostancı|arXiv (Cornell University)|Apr 19, 2016

Advanced Statistical Methods and Models1 references3 citations

TL;DR

This study compares Back Propagation Neural Networks (BPNN) and Instance-Based Learning (IBL) for classification tasks using multiple datasets and performance metrics. Results show BPNN outperforms IBL, especially with optimal parameter tuning, and Multiple Imputation handles missing values more effectively than mean/mode replacement, though it is less suitable for small datasets.

ABSTRACT

The paper presents a comparative study of the performance of Back Propagation and Instance Based Learning Algorithm for classification tasks. The study is carried out by a series of experiments will all possible combinations of parameter values for the algorithms under evaluation. The algorithm's classification accuracy is compared over a range of datasets and measurements like Cross Validation, Kappa Statistics, Root Mean Squared Value and True Positive vs False Positive rate have been used to evaluate their performance. Along with performance comparison, techniques of handling missing values have also been compared that include Mean or Mode replacement and Multiple Imputation. The results showed that parameter adjustment plays vital role in improving an algorithm's accuracy and therefore, Back Propagation has shown better results as compared to Instance Based Learning. Furthermore, the problem of missing values was better handled by Multiple imputation method, however, not suitable for less amount of data.

Motivation & Objective

• To evaluate and compare the classification performance of Back Propagation Neural Networks (BPNN) and Instance-Based Learning (IBL) algorithms.
• To investigate the impact of parameter tuning on the accuracy of both BPNN and IBL across diverse datasets.
• To assess the effectiveness of different missing value imputation techniques—mean/mode replacement versus Multiple Imputation—on classification performance.
• To analyze performance using multiple metrics including cross-validation, Kappa statistics, RMSE, and true/false positive rates.
• To determine the suitability of Multiple Imputation for small versus large datasets in classification tasks.

Proposed method

• Conducted experiments using all possible combinations of parameter values for both BPNN and IBL to assess their performance across multiple datasets.
• Applied 10-fold cross-validation to evaluate model generalization and stability.
• Used Kappa statistics to measure inter-rater agreement and model reliability beyond simple accuracy.
• Calculated Root Mean Squared Error (RMSE) to quantify prediction error in classification outcomes.
• Evaluated model performance using True Positive Rate (TPR) and False Positive Rate (FPR) to assess sensitivity and specificity.
• Implemented two missing value handling techniques: mean/mode replacement and Multiple Imputation, comparing their impact on model accuracy.

Experimental results

Research questions

• RQ1 How does Back Propagation Neural Network compare to Instance-Based Learning in terms of classification accuracy across diverse datasets?
• RQ2 To what extent does parameter tuning influence the performance of BPNN and IBL in classification tasks?
• RQ3 Which missing value imputation method—mean/mode replacement or Multiple Imputation—yields better classification results?
• RQ4 How do performance metrics like Kappa, RMSE, TPR, and FPR vary between BPNN and IBL under different parameter settings?
• RQ5 Is Multiple Imputation more effective for larger datasets compared to smaller ones in the context of classification modeling?

Key findings

• Back Propagation Neural Networks demonstrated superior classification accuracy compared to Instance-Based Learning, particularly when parameters were optimally tuned.
• Parameter adjustment was found to play a vital role in enhancing the performance of both algorithms, with BPNN benefiting more significantly from such tuning.
• Multiple Imputation outperformed mean or mode replacement in handling missing values, especially in larger datasets, leading to more robust classification outcomes.
• The Multiple Imputation method was less effective for smaller datasets, indicating a dependency on data volume for optimal performance.
• Kappa statistics and RMSE values further confirmed that BPNN consistently achieved higher reliability and lower error rates than IBL across tested configurations.
• True Positive and False Positive rates indicated that BPNN maintained better balance between sensitivity and specificity, especially in complex datasets.