[Paper Review] Building a Large Japanese Web Corpus for Large Language Models

Naoaki Okazaki, Kakeru Hattori+8 more|arXiv (Cornell University)|Apr 27, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper presents a large-scale, high-quality Japanese web corpus of 312.1 billion characters (173 million pages) extracted and refined from 21 Common Crawl snapshots (2020–2023). By applying language-specific cleaning and deduplication techniques, the corpus enables consistent 6.6–8.1 point gains in Japanese benchmark performance across multiple LLMs, including state-of-the-art results on Llama 2 13B and Mixtral 8x7B Instruct.

ABSTRACT

Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Motivation & Objective

• Address the lack of high-quality, Japanese-specific training data for open-source LLMs, which are often derived from multilingual corpora with suboptimal Japanese text quality.
• Construct a large-scale, high-fidelity Japanese web corpus from raw Common Crawl data to support effective pre-training of Japanese LLMs.
• Demonstrate the effectiveness of the corpus through continual pre-training on diverse base LLMs, achieving consistent performance gains on Japanese benchmarks.
• Establish a reproducible, publicly released corpus and fine-tuned models to support future research in Japanese NLP and responsible LLM development.

Proposed method

• Extracted raw text from 21 Common Crawl snapshots (2020–2023), totaling ~63.4 billion pages, using trafilatura for web text extraction.
• Applied MinHash-based deduplication with a Jaccard coefficient threshold of ~0.9 to remove near-duplicate content at the character 5-gram level.
• Implemented a language-specific quality filter tailored for Japanese, detecting and removing low-quality content such as excessive repetition, emoji density, and function word scarcity.
• Used a custom Japanese text quality estimator to filter out low-likelihood or malformed text, improving overall corpus coherence and utility.
• Performed continual pre-training on five base LLMs (Llama 2 7B, 13B, 70B; Mistral 7B v0.1; Mixtral 8x7B Instruct) using the refined corpus.
• Evaluated performance on standard Japanese benchmark datasets to quantify improvements in downstream NLP tasks.

Figure 1: The pipeline for building a large Japanese web corpus.

Experimental results

Research questions

• RQ1 Can a large-scale, high-quality Japanese web corpus be effectively constructed from raw Common Crawl data using language-specific filtering and deduplication?
• RQ2 To what extent does continual pre-training on the proposed corpus improve performance on Japanese NLP benchmarks across diverse LLM architectures?
• RQ3 How does the performance gain from the proposed corpus compare to existing corpora like mC4, CC-100, and OSCAR on Japanese tasks?
• RQ4 Does the inclusion of the refined Japanese corpus degrade performance in cross-lingual tasks, such as Japanese-English translation?
• RQ5 Can the corpus be used to train LLMs from scratch, and what are the implications for model generalization and safety?

Key findings

• The proposed corpus contains 312.1 billion characters (173 million pages), making it the largest available training corpus for Japanese LLMs, surpassing mC4 (239.7B), CC-100 (25.8B), and OSCAR (74B).
• Continual pre-training on the corpus yielded consistent performance gains of 6.6–8.1 points across multiple Japanese benchmark datasets, with Llama 2 13B showing the largest improvement among all compared corpora.
• The corpus achieved state-of-the-art results on all evaluated model sizes (7B, 13B, 8x7B, and 70B), confirming its effectiveness across different model scales.
• The improvement on Llama 2 13B from this corpus was larger than that from any other existing corpus, indicating its superior quality and relevance for Japanese NLP.
• The corpus enhanced performance in English-Japanese translation but slightly degraded performance in the reverse (Japanese-English) direction, suggesting a need for balanced data curation.
• All fine-tuned models and benchmark datasets are publicly released on Hugging Face, enabling full reproducibility of the experimental results.