[Paper Review] Multilinear Compressive Learning with Prior Knowledge

Dat Thanh Tran, Moncef Gabbouj+1 more|arXiv (Cornell University)|Feb 17, 2020

Sparse and Compressive Sensing Techniques49 references4 citations

TL;DR

This paper proposes a knowledge transfer framework for Multilinear Compressive Learning (MCL) that discovers discriminative tensor subspaces via supervised training of a nonlinear compressive model and transfers this knowledge to optimize sensing and feature synthesis. The method enables end-to-end learning with improved performance, especially in semi-supervised settings where unlabeled data enhances model generalization and accuracy.

ABSTRACT

The recently proposed Multilinear Compressive Learning (MCL) framework combines Multilinear Compressive Sensing and Machine Learning into an end-to-end system that takes into account the multidimensional structure of the signals when designing the sensing and feature synthesis components. The key idea behind MCL is the assumption of the existence of a tensor subspace which can capture the essential features from the signal for the downstream learning task. Thus, the ability to find such a discriminative tensor subspace and optimize the system to project the signals onto that data manifold plays an important role in Multilinear Compressive Learning. In this paper, we propose a novel solution to address both of the aforementioned requirements, i.e., How to find those tensor subspaces in which the signals of interest are highly separable? and How to optimize the sensing and feature synthesis components to transform the original signals to the data manifold found in the first question? In our proposal, the discovery of a high-quality data manifold is conducted by training a nonlinear compressive learning system on the inference task. Its knowledge of the data manifold of interest is then progressively transferred to the MCL components via multi-stage supervised training with the supervisory information encoding how the compressed measurements, the synthesized features, and the predictions should be like. The proposed knowledge transfer algorithm also comes with a semi-supervised adaption that enables compressive learning models to utilize unlabeled data effectively. Extensive experiments demonstrate that the proposed knowledge transfer method can effectively train MCL models to compressively sense and synthesize better features for the learning tasks with improved performances, especially when the complexity of the learning task increases.

Motivation & Objective

• Address the challenge of identifying discriminative tensor subspaces that capture essential features for downstream learning tasks in multilinear compressive sensing.
• Optimize the sensing and feature synthesis components in MCL to project signals onto the discovered data manifold.
• Enable effective use of unlabeled data through a semi-supervised adaptation of the knowledge transfer process.
• Improve MCL performance by transferring prior knowledge from a teacher model to a student model in an end-to-end trainable framework.
• Demonstrate the effectiveness of the approach on complex learning tasks where traditional methods struggle due to limited labeled data.

Proposed method

• Train a nonlinear compressive learning model (teacher) on the inference task to implicitly discover a high-quality data manifold.
• Transfer knowledge from the teacher model to the MCL system via multi-stage supervised training using supervisory signals for compressed measurements, features, and predictions.
• Introduce a semi-supervised adaptation that incorporates unlabeled data by applying a self-labeling procedure to enhance prior knowledge generation.
• Use the prior-generating model (P) and its semi-supervised variant (P-S) to provide structured inductive bias for the MCL student models.
• Optimize the entire system end-to-end, jointly learning sensing matrices and feature synthesis components guided by the transferred knowledge.
• Leverage the hierarchical structure of tensor data to preserve multidimensional signal structure during compression and learning.

Experimental results

Research questions

• RQ1 How can we effectively discover a discriminative tensor subspace that captures essential features for the learning task in multilinear compressive learning?
• RQ2 How can prior knowledge from a trained compressive model be systematically transferred to optimize sensing and feature synthesis in MCL?
• RQ3 To what extent does incorporating unlabeled data improve the performance of MCL models in semi-supervised settings?
• RQ4 Can knowledge transfer from a teacher model significantly enhance the generalization and accuracy of MCL systems compared to standard training?
• RQ5 How does the proposed method scale with increasing complexity of the learning task and limited labeled data?

Key findings

• The proposed MCLwP-S model, which uses semi-supervised knowledge transfer, achieves the highest performance across all benchmarks, outperforming both MCL and MCLwP.
• On the CIFAR-10S dataset with 14×11×2 measurements, MCLwP-S achieved 78.03% accuracy, surpassing MCL (75.31%) and MCLwP (75.86%).
• In the CelebA-500S semi-supervised setting, MCLwP-S reached 58.34% accuracy, significantly outperforming MCL (41.47%) and MCLwP (42.07%).
• The prior-generating model P-S, trained with unlabeled data, achieved 79.70% accuracy on CIFAR-10S, demonstrating improved knowledge extraction compared to P (76.99%).
• The performance gap between MCLwP and MCL highlights the importance of prior knowledge transfer, as MCLwP underperforms MCL in half the settings when labeled data is scarce.
• The semi-supervised adaptation enhances both the prior model and the student model, proving that unlabeled data effectively improves the quality of transferred knowledge in MCL.