[Paper Review] A Channel Coding Benchmark for Meta-Learning

Rui Li, Ondrej Bohdal+5 more|arXiv (Cornell University)|Jul 15, 2021

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper introduces MetaCC, a novel channel coding benchmark for meta-learning that enables controlled study of task distribution breadth and shift—critical challenges in real-world meta-learning. By modeling diverse wireless channel conditions as meta-training tasks, the benchmark demonstrates that meta-learners can reduce error rates by 58% over standard neural decoders and 30% over Viterbi decoding on real-world radio channels, proving practical value and systematic evaluation capability.

ABSTRACT

Meta-learning provides a popular and effective family of methods for data-efficient learning of new tasks. However, several important issues in meta-learning have proven hard to study thus far. For example, performance degrades in real-world settings where meta-learners must learn from a wide and potentially multi-modal distribution of training tasks; and when distribution shift exists between meta-train and meta-test task distributions. These issues are typically hard to study since the shape of task distributions, and shift between them are not straightforward to measure or control in standard benchmarks. We propose the channel coding problem as a benchmark for meta-learning. Channel coding is an important practical application where task distributions naturally arise, and fast adaptation to new tasks is practically valuable. We use our MetaCC benchmark to study several aspects of meta-learning, including the impact of task distribution breadth and shift, which can be controlled in the coding problem. Going forward, MetaCC provides a tool for the community to study the capabilities and limitations of meta-learning, and to drive research on practically robust and effective meta-learners.

Motivation & Objective

• To address the lack of systematic evaluation tools for meta-learning under complex, multi-modal task distributions and distribution shift—common in real-world applications.
• To provide a benchmark that enables precise measurement and control of task distribution breadth and shift, overcoming limitations of vision-based benchmarks.
• To evaluate the practical impact of meta-learning in real-world wireless communication by testing on a software-defined radio (SDR) channel.
• To enable systematic study of meta-learner robustness and generalization across diverse, realistic task distributions.
• To complement existing benchmarks by offering a scalable, elastic, and application-driven framework for meta-learning research.

Proposed method

• The benchmark models wireless channel conditions (e.g., AWGN, Rayleigh, Rician, Ricean, SDR-based) as distinct meta-training and meta-testing tasks.
• Each task corresponds to a unique channel parameter set (e.g., SNR, fading profile), enabling controlled variation in task distribution breadth and shift.
• Meta-learners are trained to rapidly adapt a neural decoder to new, unseen channel conditions using few-shot adaptation.
• The authors introduce novel metrics: task distribution breadth (measuring diversity of training tasks) and distribution shift (measuring divergence between meta-train and meta-test tasks).
• Performance is evaluated using bit error rate (BER) across synthetic and real-world channels, including a software-defined radio (SDR) measurement.
• Baselines include standard neural decoders, Viterbi decoding, and standard ERM, with meta-learners compared on adaptation speed and final error rate.

Experimental results

Research questions

• RQ1 How do existing meta-learners perform when trained on broad, multi-modal task distributions, and are they vulnerable to under-fitting?
• RQ2 How robust are meta-learners to distribution shift between meta-training and meta-testing task distributions?
• RQ3 Can meta-learned decoders achieve significant error rate reduction on real-world wireless channels compared to standard and classic codecs?
• RQ4 How does the number of unique training tasks affect meta-learner performance, especially in sparse or complex regimes?
• RQ5 To what extent does meta-learning improve generalization when shifting from simulation to real-world channel conditions?

Key findings

• Meta-learners show clear degradation in performance as task distribution breadth increases, confirming vulnerability to under-fitting on complex, multi-modal task distributions.
• Despite absolute performance drop under distribution shift, meta-learners maintain or improve their relative advantage over ERM, indicating robustness in relative performance gain.
• On a real-world SDR-based channel, meta-learned decoders achieved a 58% reduction in bit error rate compared to standard neural decoders and a 30% reduction compared to Viterbi decoding.
• Performance degradation with fewer training tasks is most pronounced for Reptile, while CAVIA and MAML FO show more stable performance under sparse task regimes.
• The benchmark reveals that meta-learners like BOIL, MetaSGD, and MetaCurvature are more sensitive to task sparsity than CAVIA and MAML FO, indicating varying robustness to low-shot adaptation.
• The study confirms that meta-learning can provide substantial practical gains in real-world communication systems, especially when rapid adaptation to new channel conditions is required.