[Paper Review] Manifoldron: Direct Space Partition via Manifold Discovery

Dayang Wang, Fenglei Fan+7 more|arXiv (Cornell University)|Jan 14, 2022

Topological and Geometric Data Analysis4 citations

TL;DR

Manifoldron introduces a non-parameterized machine learning model that directly partitions input space by discovering underlying data manifolds through Delaunay triangulation, simplex trimming, and envelope detection. It achieves competitive performance on classification and regression tasks, outperforming or matching mainstream models like MLP and SVM, especially in low-dimensional and structured data settings.

ABSTRACT

A neural network with the widely-used ReLU activation has been shown to partition the sample space into many convex polytopes for prediction. However, the parameterized way a neural network and other machine learning models use to partition the space has imperfections, extit{e}. extit{g}., the compromised interpretability for complex models, the inflexibility in decision boundary construction due to the generic character of the model, and the risk of being trapped into shortcut solutions. In contrast, although the non-parameterized models can adorably avoid or downplay these issues, they are usually insufficiently powerful either due to over-simplification or the failure to accommodate the manifold structures of data. In this context, we first propose a new type of machine learning models referred to as Manifoldron that directly derives decision boundaries from data and partitions the space via manifold structure discovery. Then, we systematically analyze the key characteristics of the Manifoldron such as manifold characterization capability and its link to neural networks. The experimental results on 4 synthetic examples, 20 public benchmark datasets, and 1 real-world application demonstrate that the proposed Manifoldron performs competitively compared to the mainstream machine learning models. We have shared our code in \url{https://github.com/wdayang/Manifoldron} for free download and evaluation.

Motivation & Objective

• Address the interpretability and rigidity issues of parameterized models like deep neural networks, which rely on fixed decision boundary forms and are hard to interpret at scale.
• Overcome the limitations of non-parameterized models (e.g., k-NN, Naive Bayes) that lack expressive power due to oversimplification or failure to model intrinsic data manifolds.
• Develop a new class of models that directly derive decision boundaries from data structure, enabling both interpretability and strong generalization via manifold-aware space partitioning.
• Bridge the conceptual gap between neural networks and geometric space partitioning by showing that both partition space into polytopes, but Manifoldron does so explicitly and non-parametrically.

Proposed method

• Apply Delaunay triangulation to the training data of each class to create an initial geometric mesh of the data manifold.
• Perform simplex trimming using a k-nearest neighbor (k-NN) strategy to remove spurious or outlier-affected simplices and retain only those forming a compact, true manifold envelope.
• Construct decision boundaries as the outer envelopes of the trimmed simplices for each class, enabling classification via in-out simplex detection and distance estimation.
• Extend the framework to regression by using a weighted average over the closest simplices in the trimmed mesh, based on proximity to the query point.
• Introduce feature bagging and point2point projection with parallel computing to reduce computational cost during training and inference.
• Leverage KD-trees for efficient neighbor search during trimming and inference, improving scalability to moderate-sized datasets.

Figure 1 : (a) The pipeline of the proposed Manifoldron. In the training stage, Delaunay triangulation, simplex trimming, and envelope detection are sequentially performed to identify the manifold structure of data and establish decision boundaries. In the test stage, we classify new points based on

Experimental results

Research questions

• RQ1 Can a machine learning model directly derive decision boundaries from the intrinsic manifold structure of data, bypassing parameterized function approximation?
• RQ2 How does the performance of such a manifold-driven model compare to mainstream parameterized models (e.g., MLP, SVM, Random Forest) in terms of accuracy and efficiency?
• RQ3 To what extent can the Manifoldron capture complex, non-linear data manifolds while maintaining interpretability and robustness?
• RQ4 What is the relationship between the space partitioning mechanism of Manifoldron and that of ReLU-based neural networks, particularly in terms of polytopic partitioning?
• RQ5 How sensitive is the Manifoldron’s performance to hyperparameters like the number of neighbors used in trimming, and can this sensitivity be mitigated?

Key findings

• On 20 public benchmark datasets, Manifoldron achieved performance on par with or better than state-of-the-art models such as MLP, SVM, and XGBoost, particularly excelling in low-dimensional and structured data.
• In regression tasks on six elementary functions, Manifoldron achieved the lowest mean squared error (MSE) on four out of six functions, with the best MSE of 0.006 on $f_2$, outperforming KNNR and GPR.
• The model demonstrated strong generalization on a real-world biological dataset (gene expression profiling), showing practical utility beyond synthetic and benchmark tasks.
• The Manifoldron achieved the lowest MSE on $f_1$ (0.101) and $f_3$ (0.095), and was second-best on $f_4$ (0.975), only surpassed by MLP, indicating strong fitting capability.
• Despite its non-parametric nature, the Manifoldron achieved acceptable inference times on datasets with tens of thousands of samples and hundreds of features, thanks to feature bagging and parallelized computation.
• The model’s performance was sensitive to the number of neighbors used in trimming, highlighting a key hyperparameter challenge for future robustness improvements.

Figure 2 : The scheme of proving the universal boundary approximation theorem.