[論文レビュー] Penalizing Localized Dirichlet Energies in Low Rank Tensor Products

Paris A. Karakasis, Nicholas D. Sidiropoulos|arXiv (Cornell University)|Jan 20, 2026

Tensor decomposition and applications被引用数 0

ひとこと要約

The paper analyzes low-rank tensor-product B-spline models (TPBS), derives a closed-form Dirichlet energy, introduces local Dirichlet energy regularization, proposes inference with incomplete data, and shows TPBS often outperform neural nets in overfitting scenarios while remaining robust to regularization.

ABSTRACT

We study low-rank tensor-product B-spline (TPBS) models for regression tasks and investigate Dirichlet energy as a measure of smoothness. We show that TPBS models admit a closed-form expression for the Dirichlet energy, and reveal scenarios where perfect interpolation is possible with exponentially small Dirichlet energy. This renders global Dirichlet energy-based regularization ineffective. To address this limitation, we propose a novel regularization strategy based on local Dirichlet energies defined on small hypercubes centered at the training points. Leveraging pretrained TPBS models, we also introduce two estimators for inference from incomplete samples. Comparative experiments with neural networks demonstrate that TPBS models outperform neural networks in the overfitting regime for most datasets, and maintain competitive performance otherwise. Overall, TPBS models exhibit greater robustness to overfitting and consistently benefit from regularization, while neural networks are more sensitive to overfitting and less effective in leveraging regularization.

研究の動機と目的

• Motivate regularization for high-capacity models to improve generalization under limited data.
• Characterize the Dirichlet energy of low-rank tensor product models and reveal issues with global energy regularization.
• Propose a localized Dirichlet energy (LDE) regularizer centered at training points to promote smoothness where data lie.
• Develop inference strategies for incomplete data leveraging TPBS structure.
• Empirically compare TPBS with neural networks across regression and classification tasks and assess overfitting behavior.

提案手法

• Show that the Dirichlet energy DE(g) of a TPBS model admits a closed-form expression DE(g)=s(g)^T Z(g) s(g).
• Define training-point-centered localized Dirichlet energy LDE_rho(g)= sum_m ∫_{B_rho(x_m)} ||∇g(x)||_F^2 dx and use it as a regularizer in empirical risk minimization.
• Use a regularization schedule that increases λ over training (λ_{t+1}=h λ_t) and select models based on validation performance, including an overfitting-aware choice.
• Provide two marginalization-based inference strategies for incomplete observations: (i) simple marginalization over unobserved features (Eq. 10) and (ii) a marginalized estimator using a low-rank density model (Eq. 12).
• Accommodate incomplete data without imputation by exploiting the TPBS factorization for efficient marginalization and inference.

Figure 1 : Double descent of test error with respect to network width for varying training set sizes on MNIST.

実験結果

リサーチクエスチョン

• RQ1 Can the Dirichlet energy of low-rank tensor-product models be computed in closed form, and what does this imply for interpolation and regularization?
• RQ2 Does a localized Dirichlet energy regularizer focusing on training-point neighborhoods improve generalization and robustness to overfitting compared to global Dirichlet energy?
• RQ3 How can TPBS models perform inference with incomplete data, and how do these methods compare to neural networks under missing data?
• RQ4 Do TPBS models maintain competitive performance across classification and regression tasks and exhibit robustness to overfitting relative to neural networks?

主な発見

Dataset	NN (best val)	NN (overfit)	TPBS (best val)	TPBS (overfit)
Ion	0.949 ± 0.026 (NR)	0.952 ± 0.014 (R)	0.930 ± 0.029 (R)	0.938 ± 0.010 (R)
BCW	0.975 ± 0.007 (NR)	0.958 ± 0.012 (NR)	0.965 ± 0.014 (R)	0.965 ± 0.014 (R)
Diabetes	0.101 ± 0.010 (R)	0.223 ± 0.037 (R)	0.134 ± 0.005 (R)	0.173 ± 0.019 (R)
Yacht	0.001 ± 0.000 (R)	0.001 ± 0.000 (R)	0.001 ± 0.000 (R)	0.001 ± 0.000 (R)
Physico	0.263 ± 0.010 (NR)	0.533 ± 0.075 (NR)	0.340 ± 0.012 (R)	0.336 ± 0.013 (R)
Sarcos	0.049 ± 0.001 (NR)	0.060 ± 0.005 (NR)	0.240 ± 0.042 (R)	0.243 ± 0.046 (R)

• Dirichlet energy for TPBS models is computable in closed form as DE(g)=s(g)^T Z(g) s(g).
• Localized Dirichlet energy regularization (LDE_rho) focuses on the data-support region and improves generalization, especially under overfitting.
• Across six datasets, LDE-regularized TPBS often outperforms or matches neural networks, with TPBS showing stronger resistance to overfitting and consistent benefit from regularization.
• In incomplete-data scenarios, TPBS with localized regularization and marginalization-based estimators remain competitive and often superior to NN baselines, particularly under overfitting.
• Regularization consistently benefits TPBS, while neural networks are more sensitive to overfitting and show less consistent gains from regularization.