[Paper Review] The representation landscape of few-shot learning and fine-tuning in large language models

Diego Doimo, Alessandro Serra+2 more|arXiv (Cornell University)|Sep 5, 2024

Speech Recognition and Synthesis4 citations

TL;DR

This paper investigates how in-context learning (ICL) and supervised fine-tuning (SFT) shape internal representations in large language models (LLMs) using a density-based geometric analysis of hidden representations. It reveals that ICL creates semantically organized, hierarchical representations in early layers, while SFT sharpens answer-identity clustering in later layers—indicating fundamentally different computational strategies despite similar performance outcomes.

ABSTRACT

In-context learning (ICL) and supervised fine-tuning (SFT) are two common strategies for improving the performance of modern large language models (LLMs) on specific tasks. Despite their different natures, these strategies often lead to comparable performance gains. However, little is known about whether they induce similar representations inside LLMs. We approach this problem by analyzing the probability landscape of their hidden representations in the two cases. More specifically, we compare how LLMs solve the same question-answering task, finding that ICL and SFT create very different internal structures, in both cases undergoing a sharp transition in the middle of the network. In the first half of the network, ICL shapes interpretable representations hierarchically organized according to their semantic content. In contrast, the probability landscape obtained with SFT is fuzzier and semantically mixed. In the second half of the model, the fine-tuned representations develop probability modes that better encode the identity of answers, while the landscape of ICL representations is characterized by less defined peaks. Our approach reveals the diverse computational strategies developed inside LLMs to solve the same task across different conditions, allowing us to make a step towards designing optimal methods to extract information from language models.

Motivation & Objective

• To understand how in-context learning (ICL) and supervised fine-tuning (SFT) shape the internal representation space of large language models (LLMs).
• To compare the geometric structure of hidden representations under ICL and SFT for the same question-answering task.
• To investigate whether ICL and SFT induce similar or divergent internal computational strategies despite comparable performance.
• To analyze the evolution of representation geometry across layers, particularly around a critical transition point near the middle of the network.
• To identify how intrinsic dimensionality and probability mode structures correlate with semantic content and answer identity.

Proposed method

• A density-based approach is used to identify probability modes directly on the data manifold in hidden layer representations, avoiding explicit dimensionality reduction.
• The intrinsic dimension (ID) of hidden representations is computed across layers to detect structural transitions in the representation landscape.
• The method tracks the emergence of multimodal structures in representations, with a focus on semantic clustering (by subject) and answer identity clustering (by letter).
• Density peaks are identified using a non-parametric method that does not assume convex cluster shapes or predefine the number of clusters.
• The analysis is applied to a multiple-choice question-answering task using the MMLU benchmark, comparing zero-shot, few-shot (ICL), and fine-tuned setups.
• The approach is validated on a second dataset mixture with similar subject partitioning, showing robustness to data distribution.

(a) Consistency between the peak composition and the MMLU subjects. A schematic view of the density peaks in the early . The coloring reflects the presence of the subjects.

Experimental results

Research questions

• RQ1 Do in-context learning and supervised fine-tuning produce similar internal representations in large language models?
• RQ2 How does the geometric structure of hidden representations evolve across layers under ICL and SFT?
• RQ3 What is the role of the intrinsic dimension in identifying functional transitions within the network?
• RQ4 Do early layers under ICL exhibit hierarchical, semantically meaningful clustering of representations?
• RQ5 Do later layers under SFT show sharper, answer-identity-specific clustering in the representation landscape?

Key findings

• A sharp transition in representation geometry occurs near the middle of the network for both ICL and SFT, marked by a peak in intrinsic dimension.
• In early layers, ICL produces semantically organized, hierarchical representations that cluster by subject, with density peaks reflecting underlying MMLU topics.
• In contrast, SFT produces a fuzzier, semantically mixed representation landscape in early layers, lacking clear subject-based clustering.
• In later layers, SFT develops well-defined probability modes that strongly correlate with answer identities (A, B, C, D), indicating enhanced answer-specific encoding.
• ICL representations in later layers show less defined peaks, suggesting weaker identity encoding compared to SFT.
• The analysis reveals that SFT modifies later layers more significantly than early layers, supporting adaptive low-rank fine-tuning strategies that prioritize later layers.

(b) Consistency between the peak composition and the MMLU answers. A schematic view of the density peaks in the late layers. The coloring reflects the presence of answers with the same letter: A (blue), B (okra), C (pink), and D (light blue).