[Paper Review] Assessing Dataset Quality Through Decision Tree Characteristics in Autoencoder-Processed Spaces

Szymon Mazurek, Maciej Wielgosz|arXiv (Cornell University)|Jun 27, 2023

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a novel framework for assessing dataset quality in machine learning classification tasks by analyzing decision tree complexity in autoencoder-processed data spaces. It demonstrates that high-quality datasets yield significantly simpler decision trees after autoencoding, while poor-quality datasets show minimal reduction in tree complexity, revealing autoencoder efficacy as a proxy for data quality.

ABSTRACT

In this paper, we delve into the critical aspect of dataset quality assessment in machine learning classification tasks. Leveraging a variety of nine distinct datasets, each crafted for classification tasks with varying complexity levels, we illustrate the profound impact of dataset quality on model training and performance. We further introduce two additional datasets designed to represent specific data conditions - one maximizing entropy and the other demonstrating high redundancy. Our findings underscore the importance of appropriate feature selection, adequate data volume, and data quality in achieving high-performing machine learning models. To aid researchers and practitioners, we propose a comprehensive framework for dataset quality assessment, which can help evaluate if the dataset at hand is sufficient and of the required quality for specific tasks. This research offers valuable insights into data assessment practices, contributing to the development of more accurate and robust machine learning models.

Motivation & Objective

• To develop a practical, data-driven framework for evaluating dataset quality in machine learning classification tasks.
• To investigate how dataset quality influences the complexity of decision trees trained on raw versus autoencoder-processed data.
• To identify whether autoencoder compression can serve as a diagnostic tool for detecting low-quality or noisy datasets.
• To evaluate the impact of feature relevance, data redundancy, and entropy on decision tree structure and model performance.
• To provide a scalable, interpretable method for practitioners to assess whether a dataset is sufficient and high-quality for a given classification task.

Proposed method

• Employed nine benchmark and custom datasets with varying quality levels, including high-quality (e.g., MNIST, CIFAR10), low-quality (e.g., expert, pitbull), high-entropy (random Gaussian), and high-redundancy (augmented CIFAR10) variants.
• Applied a deep autoencoder to compress all datasets into lower-dimensional latent representations while preserving class structure.
• Trained binary decision tree classifiers on both raw input data and autoencoder-processed latent representations.
• Quantified decision tree complexity using depth, number of nodes, and number of splits to assess structural differences.
• Used standard metrics like Gini impurity and information gain to guide tree construction and ensure consistent splitting behavior.
• Compared tree complexity across datasets and data representations to identify patterns linking data quality to model interpretability and structure.

Experimental results

Research questions

• RQ1 How does dataset quality affect the complexity of decision trees trained on raw versus autoencoder-processed data?
• RQ2 To what extent does autoencoding reduce decision tree complexity in high-quality datasets compared to low-quality ones?
• RQ3 Can the discrepancy in decision tree complexity between raw and autoencoded data serve as a reliable indicator of dataset quality?
• RQ4 How do high-entropy and high-redundancy datasets affect the performance and structure of decision trees before and after autoencoding?
• RQ5 Does the presence of irrelevant or noisy features prevent autoencoders from simplifying decision trees, and if so, why?

Key findings

• High-quality datasets, such as MNIST and CIFAR10, consistently produced simpler decision trees when trained on autoencoder-processed representations compared to raw data.
• In contrast, low-quality datasets like the 'expert' and 'pitbull' datasets showed minimal reduction in decision tree complexity after autoencoding, indicating limited feature extraction capability.
• The high-entropy dataset, generated from random Gaussian distributions, produced complex decision trees on raw data and remained complex after autoencoding, reflecting the absence of meaningful patterns.
• The high-redundancy dataset, derived from augmented CIFAR10 images, exhibited similar tree complexity in both raw and autoencoded forms, suggesting autoencoders failed to remove redundant information.
• Decision tree complexity was strongly correlated with data quality: higher quality led to greater simplification post-autoencoding, while noisy or unstructured data showed little to no improvement.
• The study confirms that autoencoders act as effective dimensionality reduction tools only when the underlying data contains coherent, informative features—otherwise, they fail to simplify downstream models.