[Paper Review] What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Boseop Kim, HyoungSeok Kim+35 more|arXiv (Cornell University)|Sep 9, 2021

Topic Modeling43 references4 citations

TL;DR

This paper introduces HyperCLOVA, a 82B-parameter Korean generative pretrained transformer trained on a 560B-token Korean corpus, demonstrating state-of-the-art in-context zero-shot and few-shot performance on Korean NLP tasks. It proposes a language-specific tokenization method and integrates prompt-based learning (e.g., p-tuning) to boost performance, while introducing HyperCLOVA Studio—a GUI-based prompt engineering interface enabling non-experts to prototype AI applications with minimal ML expertise, advancing the No Code AI paradigm.

ABSTRACT

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

Motivation & Objective

• To address the lack of large-scale, non-English language models with comprehensive analysis of model size and prompt engineering effects.
• To investigate the impact of language-specific tokenization on in-context learning performance in low-resource, agglutinative languages like Korean.
• To demonstrate the feasibility of continuous prompt-based tuning (e.g., p-tuning) in large-scale in-context learning LMs.
• To enable non-experts to prototype AI systems via an interactive, GUI-based prompt engineering platform (HyperCLOVA Studio).
• To explore the potential of a No Code AI paradigm using a single large-scale LM as a foundation for diverse applications.

Proposed method

• Constructed a 560B-token Korean-centric corpus through web crawling and data refinement, focusing on linguistic and cultural relevance.
• Designed a hybrid tokenization method combining byte-level BPE with a morpheme analyzer to better handle Korean’s agglutinative morphology.
• Trained HyperCLOVA with 39B and 82B parameters using a standard GPT-3 style autoregressive language modeling objective.
• Applied continuous prompt tuning (p-tuning) to optimize in-context learning without fine-tuning the main model parameters.
• Developed HyperCLOVA Studio, a web-based interactive interface offering GUI and API access to HyperCLOVA for non-technical users.
• Integrated input gradient functionality to enable few-shot prompt encoder fine-tuning, enhancing performance with minimal data.

Experimental results

Research questions

• RQ1 How does language-specific tokenization affect in-context learning performance in large-scale non-English LMs?
• RQ2 What are the performance differences between mid-sized (39B) and large (82B) LMs in zero-shot and few-shot settings on Korean NLP tasks?
• RQ3 Can continuous prompt-based optimization methods like p-tuning be effectively applied to in-context learning in billion-scale LMs?
• RQ4 To what extent can non-experts use an interactive prompt engineering interface to build high-performing AI prototypes without model training?
• RQ5 Can a single large-scale LM enable a No Code AI paradigm that democratizes AI development across non-technical domains?

Key findings

• HyperCLOVA with 82B parameters achieves state-of-the-art in-context zero-shot and few-shot performance on multiple Korean NLP benchmarks.
• The proposed Korean-specific tokenization method significantly improves downstream task performance compared to standard BPE, especially in morphologically complex contexts.
• p-tuning boosts performance across both classification and generation tasks, outperforming standard prompt templates and few-shot baselines.
• Mid-sized HyperCLOVA models (39B) show strong few-shot generalization, indicating scalability and efficiency trade-offs are manageable.
• HyperCLOVA Studio enables non-experts to prototype AI applications with performance comparable to human experts, using minimal data and no model training.
• The integration of input gradient-based prompt tuning allows even non-experts to achieve SOTA results with few-shot examples, demonstrating practical feasibility of No Code AI.