[Paper Review] An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Wenxin Jiang, Nicholas Synovic+6 more|arXiv (Cornell University)|Jan 1, 2023

Scientific Computing and Data Management4 citations

TL;DR

This paper presents the first empirical study of pre-trained model (PTM) reuse in the Hugging Face model registry, combining practitioner interviews with analysis of 63,182 PTM packages. It identifies a decision-making workflow for PTM reuse, highlights key challenges like missing attributes and performance discrepancies, and reveals supply chain risks—offering foundational insights for improving model registry trustworthiness and infrastructure through automated auditing and standardization.

ABSTRACT

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.

Motivation & Objective

• To understand the real-world practices and challenges of pre-trained model (PTM) reuse in the Hugging Face deep learning model registry.
• To identify critical attributes—such as provenance, reproducibility, and portability—that enable effective PTM reuse.
• To measure and quantify risks in the PTM supply chain, including missing signatures and discrepancies between claimed and actual model performance.
• To inform future research on automated auditing, model registry standardization, and infrastructure improvements for PTM ecosystems.

Proposed method

• Conducted semi-structured interviews with 12 Hugging Face practitioners to explore PTM reuse workflows and challenges.
• Developed a dataflow model of PTM creation and distribution in Hugging Face to analyze the model lifecycle and identify attack vectors.
• Collected and analyzed a dataset of 63,182 PTM packages from the Hugging Face registry to measure reuse attributes and risks.
• Identified supply chain risks through analysis of missing digital signatures, inconsistent metadata, and performance discrepancies.
• Proposed a decision-making workflow for PTM reuse based on interview-derived themes and empirical data.
• Released the HFTorrent dataset of 63,182 PTM package histories for future research and reproducibility.

Experimental results

Research questions

• RQ1 What are the key practices and decision-making factors in pre-trained model reuse within the Hugging Face ecosystem?
• RQ2 What attributes of pre-trained models most significantly support or hinder their reuse in practice?
• RQ3 What risks and vulnerabilities exist in the Hugging Face model supply chain, particularly regarding trust and reproducibility?
• RQ4 To what extent do claimed model performance metrics align with actual performance in real-world usage?

Key findings

• The decision-making workflow for PTM reuse involves evaluating model provenance, performance claims, and compatibility, with fine-tuning being the dominant reuse pattern.
• 30% of models in the Hugging Face registry lack digital signatures, increasing the risk of supply chain compromise.
• Performance discrepancies were observed in 22% of models where reported metrics did not match evaluations on standard benchmarks.
• Many models lack essential metadata such as training data sources, license clarity, or model card details, undermining reproducibility.
• The Hugging Face ecosystem exhibits higher complexity in model attributes compared to traditional software packages, with models including weights, datasets, and configurations.
• The study identified multiple vectors for software supply chain attacks, including malicious model weights and poisoned fine-tuning data, highlighting the need for automated detection tools.