[Paper Review] LLaFS: When Large Language Models Meet Few-Shot Segmentation

Lanyun Zhu, Tianrun Chen+3 more|arXiv (Cornell University)|Nov 28, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

LLaFS is the first framework to leverage large language models (LLMs) for end-to-end few-shot image segmentation by using a task-specific instruction and a region-attribute table to guide segmentation via polygon outputs. It achieves state-of-the-art performance through a pseudo-sample-based curriculum pretraining strategy, significantly outperforming prior methods on multiple benchmarks.

ABSTRACT

This paper proposes LLaFS, the first attempt to leverage large language models (LLMs) in few-shot segmentation. In contrast to the conventional few-shot segmentation methods that only rely on the limited and biased information from the annotated support images, LLaFS leverages the vast prior knowledge gained by LLM as an effective supplement and directly uses the LLM to segment images in a few-shot manner. To enable the text-based LLM to handle image-related tasks, we carefully design an input instruction that allows the LLM to produce segmentation results represented as polygons, and propose a region-attribute table to simulate the human visual mechanism and provide multi-modal guidance. We also synthesize pseudo samples and use curriculum learning for pretraining to augment data and achieve better optimization. LLaFS achieves state-of-the-art results on multiple datasets, showing the potential of using LLMs for few-shot computer vision tasks.

Motivation & Objective

• To address the limitations of conventional few-shot segmentation methods that rely solely on limited, biased support images.
• To explore the potential of large language models (LLMs) as a source of rich, generalizable prior knowledge to improve segmentation performance.
• To enable text-based LLMs to directly generate segmentation masks in a few-shot setting, moving beyond auxiliary roles.
• To design a multimodal instruction mechanism that integrates visual and textual information effectively for improved guidance.
• To overcome data scarcity in few-shot learning through synthetic pseudo-sample generation and curriculum learning.

Proposed method

• A task-tailored instruction prompt is designed to explicitly define the few-shot segmentation task for the LLM, enabling it to generate segmentation outputs as polygons.
• A region-attribute correspondence table is introduced to simulate human visual cognition by linking object regions with descriptive attributes, providing fine-grained multimodal guidance.
• A pseudo-sample synthesis method generates synthetic support-query image pairs to augment the pretraining data distribution.
• A curriculum learning strategy is applied during pretraining, gradually increasing the complexity of synthetic samples to improve model convergence and optimization.
• The LLM is fine-tuned end-to-end using the synthesized data and instruction-based supervision to produce segmentation masks directly.
• Post-prediction refinement is applied to improve edge accuracy, particularly in complex or multi-object scenarios.

Experimental results

Research questions

• RQ1 Can large language models be effectively leveraged to perform few-shot image segmentation without relying on support image features alone?
• RQ2 How can a text-based LLM be adapted to understand and generate visual segmentation outputs such as polygon masks?
• RQ3 What role does multimodal in-context instruction, combining visual regions and textual attributes, play in improving segmentation accuracy?
• RQ4 To what extent can synthetic pseudo-samples and curriculum learning mitigate data scarcity in few-shot vision tasks?
• RQ5 Does end-to-end LLM-based segmentation outperform conventional feature-guided few-shot segmentation methods?

Key findings

• LLaFS achieves state-of-the-art performance on multiple few-shot segmentation benchmarks, with a reported mIoU of 74.2.
• Removing the pseudo-sample-based curriculum pretraining mechanism leads to a 10.7% drop in mIoU, demonstrating its critical role in performance.
• The ablation study shows that the fine-grained in-context instruction, including the attribute-region table and iterative refinement, contributes up to a 4.5% mIoU improvement.
• When the number of polygon vertices (M) exceeds 16, performance slightly degrades, indicating that M=16 is optimal for balancing accuracy and model complexity.
• Loss curves show that curriculum pretraining enables faster and more stable convergence during both pretraining and fine-tuning stages.
• Visualizations confirm that LLaFS produces accurate segmentation masks even for images with multiple objects, with refinement further improving edge localization.