[Paper Review] Slicing: Nonsingular Estimation of High Dimensional Covariance Matrices Using Multiway Kronecker Delta Covariance Structures

Deniz Akdemir|arXiv (Cornell University)|Apr 10, 2011

Blind Source Separation Techniques3 references3 citations

TL;DR

This paper introduces 'slicing,' a method for nonsingular estimation of high-dimensional covariance matrices by assuming a multiway Kronecker delta structure, reducing parameter count and enabling invertibility even when N << p. The approach uses multiway array algebra and Kronecker product decomposition, with simulations and gene expression data showing stable, accurate estimation and improved classification performance (11.3% misclassification rate).

ABSTRACT

Nonsingular estimation of high dimensional covariance matrices is an important step in many statistical procedures like classification, clustering, variable selection an future extraction. After a review of the essential background material, this paper introduces a technique we call slicing for obtaining a nonsingular covariance matrix of high dimensional data. Slicing is essentially assuming that the data has Kronecker delta covariance structure. Finally, we discuss the implications of the results in this paper and provide an example of classification for high dimensional gene expression data.

Motivation & Objective

• Address the challenge of singular sample covariance matrices in high-dimensional data where N << p.
• Develop a method to estimate nonsingular, invertible covariance matrices for use in classification, clustering, and variable selection.
• Leverage multiway Kronecker delta structures to drastically reduce the number of parameters in high-dimensional covariance estimation.
• Demonstrate the method's effectiveness through simulations and real-world application to gene expression data.
• Explore extensions using sparsity-inducing penalties (e.g., GLASSO) for improved variable selection and model parsimony.

Proposed method

• Assumes high-dimensional data follows an array variate normal distribution with a multiway Kronecker delta covariance structure.
• Applies multi-linear algebra and R-matrix multiplication to represent and estimate the covariance structure efficiently.
• Uses inverse Kronecker products and properties of Kronecker products to decompose the covariance matrix into separable components.
• Employs a four-step estimation algorithm: slicing the data array, estimating component covariance matrices, combining via Kronecker products, and scaling.
• Integrates the GLASSO package for sparse estimation by applying shrinkage penalties to individual Kronecker components.
• Orders variables by variance and applies GLASSO to high-variance components to enhance sparsity and interpretability.

Experimental results

Research questions

• RQ1 Can a Kronecker delta structure be effectively used to produce nonsingular covariance estimates in high-dimensional settings with N << p?
• RQ2 How does slicing compare to standard sample covariance estimation in terms of stability and accuracy under high-dimensional, low-sample-size conditions?
• RQ3 To what extent does the choice of slicing dimensions (e.g., 2×12 vs. 3×8) and variable ordering affect estimation performance?
• RQ4 Can the integration of sparsity-inducing penalties (e.g., GLASSO) improve variable selection and classification accuracy in high-dimensional data?
• RQ5 How well does the slicing method perform in real-world applications such as gene expression classification?

Key findings

• Slicing produces stable, nonsingular covariance estimates even when N << p, as demonstrated in simulations with p = 120 and N = 10, 50, 100.
• The method accurately recovers identity and block-diagonal covariance structures with Kronecker delta structure, as shown in heatmaps of true vs. estimated matrices.
• In the Alon colon cancer dataset, linear discriminant analysis using the slicing-estimated covariance matrix achieved a 11.3% misclassification rate.
• When GLASSO is applied to high-variance components after slicing, the classification error increased slightly to 12.9%, but the method revealed that high-variance genes are more correlated among themselves.
• Low-variance components showed minimal internal correlation but mild correlation with high-variance components, suggesting a hierarchical dependency structure.
• The method enables effective dimension reduction and parameter efficiency by exploiting separable covariance structures through Kronecker decomposition.