[Paper Review] EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce

Yangning Li, Shirong Ma+7 more|arXiv (Cornell University)|Aug 14, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper introduces EcomInstruct, a 2.5-million-sample instruction-tuning dataset for e-commerce, built on Chain-of-Task (CoT) tasks using atomic e-commerce data types like product info and reviews. Fine-tuning BLOOMZ with EcomInstruct yields EcomGPT, which outperforms ChatGPT in zero-shot cross-dataset and cross-task generalization on e-commerce benchmarks due to enhanced semantic understanding from CoT training.

ABSTRACT

Recently, instruction-following Large Language Models (LLMs) , represented by ChatGPT, have exhibited exceptional performance in general Natural Language Processing (NLP) tasks. However, the unique characteristics of E-commerce data pose significant challenges to general LLMs. An LLM tailored specifically for E-commerce scenarios, possessing robust cross-dataset/task generalization capabilities, is a pressing necessity. To solve this issue, in this work, we proposed the first e-commerce instruction dataset EcomInstruct, with a total of 2.5 million instruction data. EcomInstruct scales up the data size and task diversity by constructing atomic tasks with E-commerce basic data types, such as product information, user reviews. Atomic tasks are defined as intermediate tasks implicitly involved in solving a final task, which we also call Chain-of-Task tasks. We developed EcomGPT with different parameter scales by training the backbone model BLOOMZ with the EcomInstruct. Benefiting from the fundamental semantic understanding capabilities acquired from the Chain-of-Task tasks, EcomGPT exhibits excellent zero-shot generalization capabilities. Extensive experiments and human evaluations demonstrate that EcomGPT outperforms ChatGPT in term of cross-dataset/task generalization on E-commerce tasks.

Motivation & Objective

• To address the poor generalization of general-purpose LLMs on e-commerce tasks due to unique syntactic structures and dynamic entities.
• To develop a domain-specific instruction-tuning dataset that captures the complexity and diversity of e-commerce NLP tasks.
• To design and validate Chain-of-Task (CoT) tasks as atomic intermediate tasks derived from core e-commerce data types.
• To train and evaluate EcomGPT, a specialized LLM that achieves superior zero-shot generalization across diverse e-commerce datasets and tasks.
• To demonstrate that CoT-based instruction tuning significantly improves cross-dataset and cross-task generalization in vertical domains.

Proposed method

• Constructing EcomInstruct from two sources: open-source e-commerce NLP datasets and atomic CoT tasks derived from fundamental e-commerce data types (e.g., product attributes, user reviews).
• Defining Chain-of-Task (CoT) tasks as intermediate, semantically grounded subtasks that implicitly support solving final e-commerce tasks.
• Training EcomGPT by fine-tuning the BLOOMZ backbone model on the EcomInstruct dataset using instruction-tuning paradigms.
• Employing held-in evaluation and ablation studies to isolate the contribution of CoT data at dataset, task, and task paradigm levels.
• Using both automatic metrics (F1, Rouge) and human evaluation to assess zero-shot generalization across unseen datasets and tasks.
• Applying ablation analysis to evaluate the impact of removing CoT data associated with specific datasets, tasks, or task paradigms.

Experimental results

Research questions

• RQ1 Can instruction-tuning with CoT tasks derived from e-commerce data types significantly improve zero-shot generalization in e-commerce NLP tasks?
• RQ2 How does the inclusion of CoT data from different datasets affect model generalization across unseen datasets?
• RQ3 To what extent do CoT tasks from different task types (e.g., QA, NER, classification) enhance generalization to unrelated but semantically related tasks?
• RQ4 Does the task paradigm level (e.g., classification vs. generation) influence the transferability and benefit of CoT data?
• RQ5 How does EcomGPT compare to general-purpose LLMs like ChatGPT in cross-dataset and cross-task generalization on e-commerce benchmarks?

Key findings

• EcomGPT outperforms ChatGPT in cross-dataset and cross-task generalization, achieving a 65.74% average F1 on unseen datasets compared to ChatGPT’s 64.41% in ablation settings.
• The ablation study shows that removing CoT data from the Ecom dataset reduces performance by 1.02% in average F1, indicating its critical role in generalization.
• CoT tasks from the same data type (e.g., product titles in Ecom and Youku) show mutual gains, while those from different types (e.g., CCKS and JDDC) show no such benefit.
• CoT tasks related to QA exhibit the strongest generalization boost across other tasks, suggesting QA is a powerful abstraction for instruction-following models.
• At the task paradigm level, CoT tasks from classification tasks provide greater gains to other paradigms than to their own, indicating high transferability.
• Human evaluation confirms EcomGPT’s superiority in handling complex, entity-rich e-commerce text, such as generating coherent product titles from structured attributes.