[Paper Review] Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets

Deborah Pereg, Martin Villiger+2 more|arXiv (Cornell University)|Sep 28, 2022

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a low-data, computationally efficient RNN framework for few-shot learning by leveraging the asymptotic equipartition property (AEP) to identify compact, representative data subsets. It demonstrates state-of-the-art performance in image deblurring and OCT speckle suppression using only one training image, achieving high generalization and sub-minute training times on standard hardware.

ABSTRACT

The statistical supervised learning framework assumes an input-output set with a joint probability distribution that is reliably represented by the training dataset. The learner is then required to output a prediction rule learned from the training dataset's input-output pairs. In this work, we provide meaningful insights into the asymptotic equipartition property (AEP) \citep{Shannon:1948} in the context of machine learning, and illuminate some of its potential ramifications for few-shot learning. We provide theoretical guarantees for reliable learning under the information-theoretic AEP, and for the generalization error with respect to the sample size. We then focus on a highly efficient recurrent neural net (RNN) framework and propose a reduced-entropy algorithm for few-shot learning. We also propose a mathematical intuition for the RNN as an approximation of a sparse coding solver. We verify the applicability, robustness, and computational efficiency of the proposed approach with image deblurring and optical coherence tomography (OCT) speckle suppression. Our experimental results demonstrate significant potential for improving learning models' sample efficiency, generalization, and time complexity, that can therefore be leveraged for practical real-time applications.

Motivation & Objective

• To investigate the theoretical foundations of few-shot learning using the asymptotic equipartition property (AEP) in information theory.
• To address the challenge of poor generalization and high computational cost in low-data machine learning scenarios, especially in medical imaging.
• To develop a compact, efficient RNN framework that generalizes well from minimal training data, reducing training time and resource demands.
• To provide a mathematical interpretation of RNNs as sparse coding solvers, explaining their empirical success.
• To validate the method on real-world medical imaging tasks—image deblurring and OCT speckle suppression—under data and system mismatches.

Proposed method

• Theoretical analysis of the AEP is used to justify that a small, representative subset of data can effectively capture the input-output distribution, enabling reliable learning with few samples.
• A compact RNN-based architecture is proposed, trained on a single image, to perform few-shot image restoration tasks such as deblurring and speckle suppression.
• The RNN is interpreted as a sparse coding solver that iteratively refines predictions from an initial state derived from the previous time step.
• Receptive field normalization (RFN) is integrated into the RNN to reduce input data distribution entropy, improving robustness in noisy or mismatched conditions.
• The framework is trained end-to-end using a standard loss function, with convergence achieved in under 700 iterations.
• The method is applied to both synthetic and real OCT data, including cases with tissue and system mismatches, to test generalization.

Experimental results

Research questions

• RQ1 Can the asymptotic equipartition property (AEP) be leveraged to identify a minimal, representative data subset for reliable few-shot learning?
• RQ2 How can a recurrent neural network be interpreted as an approximation of a sparse coding solver in the context of few-shot learning?
• RQ3 To what extent can a single-image training setup generalize across different tissues and imaging systems in medical imaging tasks?
• RQ4 What is the computational and generalization performance of an RNN framework trained with minimal data and low computational cost?
• RQ5 Can receptive field normalization (RFN) improve robustness in noisy or mismatched data scenarios?

Key findings

• The proposed RNN framework achieves significant speckle suppression in OCT imaging with a PSNR of 31.1 dB on blueberry tissue and 31.34 dB on cucumber tissue, despite tissue and system mismatches.
• Training converges in less than 700 iterations, taking only 4.56 seconds on a laptop GPU and 35 seconds on a CPU workstation, demonstrating extreme computational efficiency.
• The method generalizes well across different tissue types and imaging systems, maintaining high visual quality and structural fidelity even under mismatched conditions.
• Theoretical analysis confirms that a small, representative data subset can reliably represent the full input-output distribution under the AEP, enabling robust learning with few samples.
• The RNN framework with RFN reduces input distribution entropy, leading to improved performance in noisy environments and enhanced visual quality in restoration tasks.
• The interpretation of RNNs as sparse coding solvers provides a mathematical intuition for their success in few-shot learning, linking their architecture to optimization behavior.