[Paper Review] Language Through a Prism: A Spectral Approach for Multiscale Language Representations

Alex Tamkin, Dan Jurafsky+1 more|arXiv (Cornell University)|Nov 9, 2020

Topic ModelingComputer Science72 references22 citations

TL;DR

This paper introduces a spectral approach using discrete cosine transforms (DCT) to disentangle multiscale linguistic structure in deep language representations. By applying spectral filters to neuron activations across input sequences, the method isolates information at different scales—word-level, utterance-level, or document-level—enabling improved performance on scale-specific tasks. The key contribution is the prism layer, which trains neurons to specialize in specific scales, enhancing long-range context modeling and outperforming BERT on multiscale NLP tasks.

ABSTRACT

Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn more about particular scales. Concretely, we apply spectral filters to the activations of a neuron across an input, producing filtered embeddings that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for training models, which uses spectral filters to constrain different neurons to model structure at different scales. Our proposed BERT + Prism model can better predict masked tokens using long-range context and produces multiscale representations that perform better at utterance- and document-level tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.

Motivation & Objective

• To understand how deep language models like BERT represent linguistic structure across multiple scales, from subwords to documents.
• To develop a general framework for isolating and controlling information at different timescales in neural network activations without relying on predefined linguistic structures.
• To improve model performance on scale-specific NLP tasks by training neurons to specialize in capturing structure at particular scales.
• To enhance long-range context modeling by constraining representations to preserve information at broader temporal scales.
• To extend spectral analysis techniques from signal processing to NLP, enabling interpretability and control over multiscale representations.

Proposed method

• Apply the discrete cosine transform (DCT) to the activation sequence of each neuron across input tokens to represent it in the frequency domain.
• Use spectral filtering (low-pass, band-pass, high-pass) to selectively remove or retain components corresponding to specific timescales in neuron activations.
• Construct scale-disentangled representations by filtering activations to emphasize either fast (word-level) or slow (document-level) variations.
• Introduce the prism layer—a learnable module that applies different spectral filters to different neurons during training, forcing them to specialize in distinct scales.
• Train a BERT-based model with the prism layer to produce multiscale representations that are more sensitive to long-range dependencies.
• Use the DCT coefficient attenuation strategy to suppress irrelevant frequency components, thereby isolating structure at target scales.

Experimental results

Research questions

• RQ1 To what extent do pre-trained language models like BERT encode linguistic structure across multiple scales?
• RQ2 Can spectral filtering be used to disentangle scale-specific information from existing neural representations without architectural changes?
• RQ3 Can we train models to specialize in capturing structure at specific scales using spectral constraints?
• RQ4 Does the prism layer improve performance on tasks requiring long-range context or multiscale understanding?
• RQ5 How do spectral filters affect the model’s sensitivity to long-range dependencies in masked language modeling?

Key findings

• Spectral filtering successfully isolates scale-specific information: low-pass filtered representations perform best on document-level topic classification, band-pass on utterance-level dialog act classification, and high-pass on word-level part-of-speech tagging.
• The BERT + Prism model achieves comparable or better performance than standard BERT across all three tasks—part-of-speech tagging, dialog act classification, and topic classification—demonstrating improved multiscale representation learning.
• The prism layer increases the model’s sensitivity to long-range context, as evidenced by improved performance on masked language modeling tasks requiring contextual reasoning over longer spans.
• Filtered representations trained with spectral filters show strong performance on their target task but degrade on others, confirming successful scale disentanglement.
• The method is generalizable beyond NLP: it applies to any model with structured representations along a temporal or spatial axis, such as in vision, audio, or time series.
• Spectral analysis provides a principled, interpretable framework for probing and controlling how deep models encode information at different scales.