[Paper Review] Higher Order Mutual Information Approximation for Feature Selection

Jilin Wu, Soumyajit Gupta+1 more|arXiv (Cornell University)|Dec 2, 2016

Face and Expression Recognition21 references3 citations

TL;DR

This paper proposes Higher Order Feature Selection (HOFS), an information-theoretic method that improves feature selection by approximating higher-order mutual information using incremental ICA, enabling the discovery of synergistic feature subsets rather than relying on first-order approximations. HOFS outperforms existing methods in capturing interdependent feature relationships while maintaining comparable computational complexity.

ABSTRACT

Feature selection is a process of choosing a subset of relevant features so that the quality of prediction models can be improved. An extensive body of work exists on information-theoretic feature selection, based on maximizing Mutual Information (MI) between subsets of features and class labels. The prior methods use a lower order approximation, by treating the joint entropy as a summation of several single variable entropies. This leads to locally optimal selections and misses multi-way feature combinations. We present a higher order MI based approximation technique called Higher Order Feature Selection (HOFS). Instead of producing a single list of features, our method produces a ranked collection of feature subsets that maximizes MI, giving better comprehension (feature ranking) as to which features work best together when selected, due to their underlying interdependent structure. Our experiments demonstrate that the proposed method performs better than existing feature selection approaches while keeping similar running times and computational complexity.

Motivation & Objective

• To address the limitation of first-order mutual information approximations in existing feature selection methods, which fail to capture multi-way feature interactions.
• To develop a method that identifies feature subsets—rather than single features—that jointly maximize mutual information with the class label.
• To maintain computational efficiency comparable to state-of-the-art methods while improving model interpretability and predictive accuracy.
• To handle heterogeneous data types (numerical and non-numerical features) effectively in real-world feature selection scenarios.
• To provide a more comprehensive understanding of feature dependencies by ranking feature subsets based on their collective information gain.

Proposed method

• HOFS uses incremental Independent Component Analysis (ICA) to approximate higher-order mutual information, capturing complex interdependencies among features.
• Instead of assuming feature independence (as in first-order approximations), HOFS models joint entropy through ICA-based decomposition of feature space.
• The method incrementally builds feature subsets by maximizing the mutual information between the subset and the class label, using a greedy selection strategy.
• Feature subsets are ranked based on their global mutual information gain, allowing identification of synergistic combinations that outperform individual features.
• The algorithm integrates ICA to estimate the joint distribution of features, avoiding the computational intractability of exact higher-order MI computation.
• HOFS is designed to be computationally efficient, with time complexity similar to existing filter-based methods, enabling scalability to high-dimensional datasets.

Experimental results

Research questions

• RQ1 Can higher-order mutual information approximation improve feature selection performance by capturing synergistic interactions among features?
• RQ2 How does HOFS compare to first-order MI-based methods in identifying relevant feature subsets, especially in the presence of redundant or correlated features?
• RQ3 Can HOFS effectively handle heterogeneous datasets with mixed numerical and non-numerical features while preserving underlying feature structures?
• RQ4 Does the proposed method maintain computational efficiency comparable to existing approaches despite modeling higher-order dependencies?
• RQ5 To what extent does HOFS improve model interpretability by revealing feature combinations that jointly contribute to prediction?

Key findings

• HOFS successfully identifies feature subsets that jointly maximize mutual information with the class label, outperforming first-order MI methods in capturing complex dependencies.
• In synthetic data with correlated and redundant features, HOFS correctly groups features like {F₁, F₆, F₂, F₇} and {F₉, F₄, F₁₄, F₁₀, F₅, F₁₅}, reflecting their true underlying structure.
• Unlike VMI (which fails under non-linear and heterogeneous conditions), HOFS maintains robust performance and correctly identifies informative feature groups even when features are biased or noisy.
• HOFS achieves better predictive performance than baseline methods while maintaining similar running times and computational complexity.
• The method effectively handles non-numerical features and preserves structural relationships in synthetic data, as demonstrated by the correct segregation of class labels based on feature groupings.
• In the heterogeneous synthetic model, HOFS produces a ranked list of feature subsets where the top subsets show high information gain, with the first subset achieving a global MI gain of approximately 0.110, closely matching the ground truth.