[Paper Review] Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models

Nguyen Quang Đuc, Le Hai Son+4 more|arXiv (Cornell University)|Mar 3, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces a comprehensive suite of open-source datasets and fine-tuned LLMs for Vietnamese natural language processing, including a 32M-article NewsCorpus, structured Sapo and category datasets, Alpaca-style instruction data, synthetic dialogues, and two Vietnamese-optimized LLaMA2-7b models pre-trained on 40GB and 120GB of curated multilingual text. The key contribution is a significant improvement in Vietnamese LLM performance through domain-specific, high-quality pretraining data and a custom tokenizer reducing token count by 70% compared to LLaMA2.

ABSTRACT

This paper presents our contributions towards advancing the state of Vietnamese language understanding and generation through the development and dissemination of open datasets and pre-trained models for Vietnamese Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs).

Motivation & Objective

• To address the lack of large-scale, high-quality training data for Vietnamese LLMs and RAG systems.
• To improve Vietnamese language understanding and generation by creating specialized datasets for different NLP tasks.
• To enhance model efficiency and performance through a custom Vietnamese tokenizer and continual pretraining on diverse, domain-specific text.
• To foster an open research ecosystem by releasing datasets and models under open licenses for community collaboration.
• To advance Vietnamese NLP by enabling state-of-the-art results in instruction tuning, semantic search, and dialogue generation.

Proposed method

• Constructed a 32M-article Vietnamese NewsCorpus (53GB) by merging and deduplicating news from Binhvq and new sources up to November 2023.
• Created the NewsSapo dataset in 'title-abstract-content' format (31.7M triples) for training sentence and passage embeddings.
• Built the NewsCategory dataset (596K samples across 21 topics) for text classification using VnExpress articles.
• Generated two Alpaca-style instruction datasets: one with 'instruction/input/output' format (50K samples) and another with 'input/output' format (25K samples), using GPT-4 and GPT-3.5.
• Developed a synthetic self-chat dataset (30K dialogues) using instruction generation and GPT-4 to improve multi-turn conversation capabilities.
• Trained a Vietnamese bi-encoder model on merged data (MSMarco, SQuAD v2, 80% of Zalo legal challenge) using PhoBERT-base-v2 as backbone, achieving 73.28% Acc@1 on the Zalo test split.

Experimental results

Research questions

• RQ1 How can large-scale, high-quality Vietnamese text data be systematically collected and curated for continual pretraining of LLMs?
• RQ2 To what extent can a domain-specific, multilingual pretraining mixture improve the performance of Vietnamese LLMs on downstream tasks?
• RQ3 Can a custom Vietnamese tokenizer significantly reduce tokenization overhead and improve model efficiency compared to standard tokenizers?
• RQ4 How effective are synthetic instruction and dialogue data in enhancing the instruction-following and conversational abilities of Vietnamese LLMs?
• RQ5 What performance gains can be achieved by pretraining a base LLM on 120GB of diverse Vietnamese text compared to 40GB?

Key findings

• The Vietnamese-LLaMA2-7b-120GB model achieved a 124GB pretraining dataset (104GB Vietnamese, 20GB English), trained over 40 days on four A100 GPUs with 4000 GPU hours.
• The custom Vietnamese tokenizer reduced token count by 70% compared to LLaMA2 and 50% compared to ChatGPT when encoding Vietnamese text.
• The Vietnamese bi-encoder model achieved 73.28% Acc@1 on the Zalo Legal Text Retrieval challenge, outperforming PhoBERT-base-v2 pre-trained on MSMarco by 25.44 percentage points.
• The 50K instruction-tuned Alpaca dataset in 'instruction/input/output' format was generated using GPT-4 and GPT-3.5, enabling diverse and high-quality supervised fine-tuning.
• The 25K 'input/output' format Alpaca dataset produced longer and more diverse outputs, improving model response quality in dialogue and instruction tasks.
• The NewsCategory dataset achieved 21-class text classification with 596,524 samples across topics like World, Business, Health, and Entertainment, enabling robust topic modeling.