[Paper Review] SamLP: A Customized Segment Anything Model for License Plate Detection

Haoxuan Ding, Junyu Gao+2 more|arXiv (Cornell University)|Jan 12, 2024

Vehicle License Plate Recognition4 citations

TL;DR

This paper proposes SamLP, the first license plate detection model based on the Segment Anything Model (SAM), leveraging LoRA for parameter-efficient fine-tuning to adapt SAM to LP detection. It further introduces a promptable fine-tuning strategy to preserve SAM’s prompt-based segmentation capability, achieving state-of-the-art performance with strong few-shot and zero-shot generalization on diverse LP datasets.

ABSTRACT

With the emergence of foundation model, this novel paradigm of deep learning has encouraged many powerful achievements in natural language processing and computer vision. There are many advantages of foundation model, such as excellent feature extraction power, mighty generalization ability, great few-shot and zero-shot learning capacity, etc. which are beneficial to vision tasks. As the unique identity of vehicle, different countries and regions have diverse license plate (LP) styles and appearances, and even different types of vehicles have different LPs. However, recent deep learning based license plate detectors are mainly trained on specific datasets, and these limited datasets constrain the effectiveness and robustness of LP detectors. To alleviate the negative impact of limited data, an attempt to exploit the advantages of foundation model is implement in this paper. We customize a vision foundation model, i.e. Segment Anything Model (SAM), for LP detection task and propose the first LP detector based on vision foundation model, named SamLP. Specifically, we design a Low-Rank Adaptation (LoRA) fine-tuning strategy to inject extra parameters into SAM and transfer SAM into LP detection task. And then, we further propose a promptable fine-tuning step to provide SamLP with prompatable segmentation capacity. The experiments show that our proposed SamLP achieves promising detection performance compared to other LP detectors. Meanwhile, the proposed SamLP has great few-shot and zero-shot learning ability, which shows the potential of transferring vision foundation model. The code is available at https://github.com/Dinghaoxuan/SamLP

Motivation & Objective

• To address the limited generalization and robustness of existing deep learning-based license plate detectors due to reliance on narrow, biased training datasets.
• To explore the potential of vision foundation models—specifically SAM—for license plate detection by customizing them for this task.
• To overcome the computational burden of fine-tuning large foundation models by employing parameter-efficient fine-tuning (LoRA).
• To preserve SAM’s prompt-based segmentation capability during adaptation, enabling future refinement with user-provided prompts.
• To evaluate the few-shot and zero-shot generalization performance of the proposed model across diverse LP styles and regions.

Proposed method

• Fine-tune the image encoder and mask decoder of SAM using Low-Rank Adaptation (LoRA), injecting low-rank matrices to efficiently adapt the foundation model to license plate detection without full fine-tuning.
• Introduce a promptable fine-tuning step after LoRA adaptation to restore and enhance SAM’s ability to generate accurate masks using point, bounding box, or mask prompts.
• Use a cascade iterative refinement mechanism during inference, where predictions are successively refined using prompts to improve segmentation accuracy.
• Select the optimal LoRA rank (r=4) through ablation to balance model size and detection performance.
• Evaluate inference performance under different prompt types (none, point, mask, and both) to determine the most effective prompting strategy.
• Apply knowledge distillation in future work to reduce inference time and improve efficiency of the foundation model-based detector.

Experimental results

Research questions

• RQ1 Can a vision foundation model like SAM be effectively adapted for license plate detection despite the diversity in LP styles and appearances across regions and vehicle types?
• RQ2 How does LoRA-based parameter-efficient fine-tuning compare to full fine-tuning in terms of detection accuracy and computational efficiency for LP detection?
• RQ3 To what extent is the prompt-based segmentation capability of SAM preserved or degraded after LoRA fine-tuning, and can it be restored through additional promptable fine-tuning?
• RQ4 What is the optimal prompting strategy (point, mask, or both) for inference in the adapted SamLP model?
• RQ5 How well does the proposed SamLP generalize in few-shot and zero-shot settings across diverse LP datasets?

Key findings

• SamLP achieves a 96.8 F1-score and 94.9 mAP on the test set when no prompts are used during inference, demonstrating strong base detection performance.
• After promptable fine-tuning, SamLP_P achieves 96.9 F1-score and 95.2 mAP using mask prompts, outperforming both the original SamLP and other LP detectors.
• The use of point prompts during inference leads to a drop in performance (to 96.3 F1-score and 94.3 mAP), likely due to error propagation from initial false predictions.
• Mask prompts provide richer context, resulting in better refinement and higher accuracy than point prompts, which are more sensitive to initial prediction errors.
• The optimal LoRA rank is r=4, as higher ranks offer no significant performance gain while increasing parameter count.
• The cascade iterative refinement with one refinement step (Num=1) achieves the best trade-off between accuracy and inference speed, with minimal computational overhead.