[Paper Review] A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

SungJun Cho, Chetan Gohil+3 more|arXiv (Cornell University)|Feb 18, 2026

Functional Brain Connectivity Studies0 citations

TL;DR

This paper systematically compares learnable and non-learnable sample-level tokenization strategies for MEG foundation models, showing simple fixed tokenization can match learnable methods in most criteria except subject fingerprinting where learnable tokens help.

ABSTRACT

Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.

Motivation & Objective

• Motivate foundation-model-style learning for MEG data and understand how tokenization shapes representations and downstream tasks.
• Systematically compare learnable and non-learnable sample-level tokenizers on MEG data across reconstruction, modeling, and decoding tasks.
• Assess whether simple fixed discretization suffices for effective neural time-series foundation modeling.

Proposed method

• Propose a learnable sample-level tokenizer tailored to MEG signals implemented as an autoencoder-like (VQ-VAE–style) framework with an encoder (GRU) and a decoder using 1D token Kernels.
• Train tokenizer variants under an annealed soft-to-hard token assignment to enable gradient flow during optimization.
• Compare with two fixed, non-learnable baselines: μ-transform and standard-quantile tokenizers.
• Pretrain a GPT-style MEG foundation model (MEG-GPT) with next-token prediction on tokenized MEG data and evaluate across multiple criteria.
• Evaluate tokenization impact on reconstruction fidelity, token prediction accuracy, biological plausibility of generated data, subject-specific information retention, and downstream decoding.

Figure 1: Overview of the foundation modeling framework and tokenizer architecture. (a) Schematic of the full generative training pipeline for the MEG-GPT foundation model. (b) Architecture of the learnable MEG tokenizer.

Experimental results

Research questions

• RQ1 Do learnable sample-level tokenizers offer meaningful gains over fixed tokenization in MEG foundation-model pipelines?
• RQ2 How does tokenization choice affect reconstruction quality and downstream tasks like zero-shot and fine-tuned decoding?
• RQ3 Is simple fixed discretization sufficient for robust MEG foundation modeling across diverse datasets and tasks?
• RQ4 Does a learnable tokenizer improve subject fingerprinting or inter-subject discriminability compared to fixed schemes?

Key findings

• Both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most criteria.
• Learnable tokenizers yield consistent improvements in subject fingerprinting compared to fixed tokenizers.
• Fixed sample-level tokenization strategies perform comparably to learnable ones on token prediction, biological plausibility, and downstream decoding in most settings.
• Across three MEG datasets with varying acquisition sites and paradigms, tokenization choices did not dramatically alter overall foundation-model performance.
• The study provides a first systematic, controlled comparison of sample-level tokenizers for MEG foundation models, with code available for replication.

Figure 2: MEG-GPT foundation model architecture. (a) High-level overview of the model architecture. (b) Detailed structure of the transformer decoder component.