[Paper Review] GPTs Window Shopping: An analysis of the Landscape of Custom ChatGPT Models

Benjamin Zi Hao Zhao, Muhammad Ikram+1 more|arXiv (Cornell University)|May 17, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This paper analyzes the landscape of custom GPTs on OpenAI's GPT Store and a third-party storefront, EpicGPTstore, using large-scale data collection and web infrastructure analysis. It reveals that English dominates GPT creation, categorization is underused, and nearly half of creators externally monetize their GPTs via websites, often promoting blogs or paid services, indicating a shift toward off-platform revenue models despite OpenAI’s promised revenue sharing.

ABSTRACT

OpenAI's ChatGPT initiated a wave of technical iterations in the space of Large Language Models (LLMs) by demonstrating the capability and disruptive power of LLMs. OpenAI has prompted large organizations to respond with their own advancements and models to push the LLM performance envelope. OpenAI has prompted large organizations to respond with their own advancements and models to push the LLM performance envelope. OpenAI's success in spotlighting AI can be partially attributed to decreased barriers to entry, enabling any individual with an internet-enabled device to interact with LLMs. What was previously relegated to a few researchers and developers with necessary computing resources is now available to all. A desire to customize LLMs to better accommodate individual needs prompted OpenAI's creation of the GPT Store, a central platform where users can create and share custom GPT models. Customization comes in the form of prompt-tuning, analysis of reference resources, browsing, and external API interactions, alongside a promise of revenue sharing for created custom GPTs. In this work, we peer into the window of the GPT Store and measure its impact. Our analysis constitutes a large-scale overview of the store exploring community perception, GPT details, and the GPT authors, in addition to a deep-dive into a 3rd party storefront indexing user-submitted GPTs, exploring if creators seek to monetize their creations in the absence of OpenAI's revenue sharing.

Motivation & Objective

• To understand the current state of the OpenAI GPT Store and the broader ecosystem of custom GPTs created by non-expert users.
• To investigate how creators are leveraging external platforms like EpicGPTstore to promote and monetize their custom GPTs.
• To assess the extent to which creators are bypassing OpenAI’s official monetization framework by directing users to external websites.
• To examine the web infrastructure and security risks associated with domains linked to GPT creators, including potential malware or phishing threats.
• To provide a public dataset and codebase to support future research on LLM customization and marketplace dynamics.

Proposed method

• Collected a dataset of 334,000 custom GPTs from the Beetrove search engine discovery system, which crawls the GPT ecosystem.
• Compared this with a curated dataset of 4,186 GPTs from the third-party EpicGPTstore, where creators proactively submitted their GPTs for indexing.
• Analyzed metadata such as GPT titles, descriptions, categories, and creation dates to identify trends in language use, categorization, and timing of creation.
• Conducted web crawling on 439 domains linked by GPT creators to detect monetization signals, including pricing terms, blogs, and subscription references.
• Performed VirusTotal scans on all collected domains to detect malicious indicators, using 92 antivirus engines to assess risk.
• Used natural language processing to detect monetization-related keywords (e.g., 'free trial', 'subscription', '$') in website content to infer external monetization intent.

Experimental results

Research questions

• RQ1 To what extent are custom GPTs on the OpenAI GPT Store created in English, and how does this compare to other languages?
• RQ2 How are creators utilizing OpenAI’s official categorization system, and what factors correlate with GPT creation spikes?
• RQ3 What proportion of GPT creators are actively monetizing their creations through external websites, and what types of monetization strategies are employed?
• RQ4 What security risks are associated with domains linked by GPT creators, such as malware or phishing indicators?
• RQ5 How do third-party storefronts like EpicGPTstore influence creator behavior and external monetization trends?

Key findings

• 89.4% of the 334,000 GPTs in the Beetrove dataset were created in English, indicating a strong linguistic dominance in the custom GPT ecosystem.
• Only 12.5% of GPTs in the Beetrove dataset were assigned to a category, suggesting widespread underuse of OpenAI’s official categorization system.
• GPT creation activity strongly correlated with two major OpenAI announcements: the GPT Store launch and the DevDay 2024 event, indicating external events drive user engagement.
• Among 396 valid domains linked by GPT creators, 193 (48.7%) contained at least one monetization-related keyword, indicating a significant shift toward off-platform revenue models.
• 132 (33.3%) of the valid domains contained blog references, suggesting creators use blogs as a promotional or content-driven monetization strategy.
• VirusTotal scans flagged 10 domains as malicious across 92 scanners, with gptjp.net and citibankdemobusiness.dev showing notable risk indicators, though overall malware prevalence was low.