[Paper Review] eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data

Bo Peng, Xinyi Ling+3 more|arXiv (Cornell University)|Feb 13, 2024

Recommender Systems and Techniques4 citations

TL;DR

This paper introduces eCeLLM, a series of instruction-tuned large language models for e-commerce, built on ECInstruct—a large-scale, high-quality, open-source instruction dataset covering 10 diverse e-commerce tasks. eCeLLM significantly outperforms both GPT-4 and task-specific models in in-domain evaluations and demonstrates strong generalization to out-of-domain scenarios, including unseen products and instructions, with a 9.3% improvement on unseen products over the best baselines.

ABSTRACT

With tremendous efforts on developing effective e-commerce models, conventional e-commerce models show limited success in generalist e-commerce modeling, and suffer from unsatisfactory performance on new users and new products - a typical out-of-domain generalization challenge. Meanwhile, large language models (LLMs) demonstrate outstanding performance in generalist modeling and out-of-domain generalizability in many fields. Toward fully unleashing their power for e-commerce, in this paper, we construct ECInstruct, the first open-sourced, large-scale, and high-quality benchmark instruction dataset for e-commerce. Leveraging ECInstruct, we develop eCeLLM, a series of e-commerce LLMs, by instruction-tuning general-purpose LLMs. Our comprehensive experiments and evaluation demonstrate that eCeLLM models substantially outperform baseline models, including the most advanced GPT-4, and the state-of-the-art task-specific models in in-domain evaluation. Moreover, eCeLLM exhibits excellent generalizability to out-of-domain settings, including unseen products and unseen instructions, highlighting its superiority as a generalist e-commerce model. Both the ECInstruct dataset and the eCeLLM models show great potential in empowering versatile and effective LLMs for e-commerce. ECInstruct and eCeLLM models are publicly accessible through https://ninglab.github.io/eCeLLM.

Motivation & Objective

• To address the limitations of task-specific e-commerce models in generalist modeling and out-of-domain generalization, especially for new users and new products.
• To bridge the gap in leveraging large language models (LLMs) for versatile e-commerce applications by creating a high-quality, large-scale instruction dataset.
• To develop a generalist e-commerce foundation model that excels across diverse tasks and generalizes effectively to unseen data.
• To enable zero-shot and few-shot generalization to new products and new instructions, overcoming cold-start challenges in e-commerce.

Proposed method

• Construct ECInstruct, a 116,528-sample, open-sourced instruction dataset spanning 10 real-world e-commerce tasks across 4 categories, with inputs, instructions, outputs, and optional options.
• Collect and curate high-quality, diverse, and realistic e-commerce instruction data from real user queries and product interactions.
• Fine-tune six general-purpose LLMs—including Llama 2 and Mistral—using ECInstruct via instruction tuning to create the eCeLLM series of models.
• Design evaluation protocols that include in-domain (IND) and out-of-domain (OOD) test sets, with OOD samples including unseen products and unseen instructions.
• Evaluate eCeLLM across multiple settings: full training, few-shot, and zero-shot, including OOD generalization to new products and new instructions.
• Use rigorous human and automatic evaluation to validate performance across diverse e-commerce tasks such as product description generation, attribute extraction, and question answering.

Figure 1 : Overall scheme of $\mathop{\mathtt{eCeLLM}}\limits$ instruction-tuned with $\mathop{\mathtt{ECInstruct}}\limits$

Experimental results

Research questions

• RQ1 Can instruction-tuned LLMs achieve superior performance across a broad spectrum of e-commerce tasks compared to task-specific models and GPT-4?
• RQ2 To what extent can eCeLLM generalize to out-of-domain settings, such as new products and new instructions, without fine-tuning?
• RQ3 How does the scale and quality of the ECInstruct dataset impact the generalization and performance of the resulting eCeLLM models?
• RQ4 Can a single generalist LLM replace multiple specialized e-commerce models while maintaining or exceeding their performance?
• RQ5 What is the impact of training data diversity and instruction quality on zero-shot and few-shot generalization in e-commerce LLMs?

Key findings

• eCeLLM models achieve an average 10.7% performance improvement over the best baseline models in in-domain evaluations across all 10 tasks.
• eCeLLM outperforms GPT-4 Turbo on all 10 tasks in the in-domain setting, demonstrating strong competitive performance.
• On out-of-domain settings with unseen products, eCeLLM achieves a 9.3% improvement over the best baseline models, highlighting its strong generalization capability.
• eCeLLM maintains strong performance in zero-shot and few-shot settings, showing robustness to distribution shift and data scarcity.
• The ECInstruct dataset enables effective zero-shot transfer to new instructions, confirming its utility for generalist modeling.
• The combination of high-quality instruction data and instruction tuning enables eCeLLM to generalize effectively to unseen tasks and product categories, solving the cold-start problem in e-commerce.

(a) IND Evaluation