[Paper Review] BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer

Zuchao Li, Shitou Zhang+3 more|arXiv (Cornell University)|Jul 1, 2023

Topic Modeling4 citations

TL;DR

BatGPT is a bidirectional autoregressive large language model developed by Wuhan University and Shanghai Jiao Tong University, designed to improve fluency, coherence, and context retention by modeling left-to-right and right-to-left dependencies. It achieves state-of-the-art performance on Chinese NLP benchmarks, scoring 36.72% average accuracy on CMMLU and 57.4% on C-Eval, with strong zero-shot reasoning on hard subsets.

ABSTRACT

BatGPT is a large-scale language model designed and trained jointly by Wuhan University and Shanghai Jiao Tong University. It is capable of generating highly natural and fluent text in response to various types of input, including text prompts, images, and audio. In the modeling level, we employ a bidirectional autoregressive architecture that allows the model to efficiently capture the complex dependencies of natural language, making it highly effective in tasks such as language generation, dialog systems, and question answering. Moreover, the bidirectional autoregressive modeling not only operates from left to right but also from right to left, effectively reducing fixed memory effects and alleviating model hallucinations. In the training aspect, we propose a novel parameter expansion method for leveraging the pre-training of smaller models and employ reinforcement learning from both AI and human feedback, aimed at improving the model's alignment performance. Overall, these approaches significantly improve the effectiveness of BatGPT, and the model can be utilized for a wide range of natural language applications.

Motivation & Objective

• To address limitations in existing autoregressive models, such as limited memory and hallucinations, by introducing a bidirectional autoregressive architecture.
• To improve alignment with human preferences and task-specific objectives through reinforcement learning from both AI and human feedback.
• To enhance performance on low-resource and complex reasoning tasks via parameter expansion and fine-tuning strategies.
• To develop a versatile, large-scale language model capable of handling text, image, and audio inputs with high fluency and contextual coherence.

Proposed method

• Employ a bidirectional autoregressive architecture that processes sequences from both left-to-right and right-to-left directions to capture long-range dependencies.
• Use a parameter expansion strategy to leverage pre-trained smaller models, enabling efficient training of the larger BatGPT-15B model.
• Integrate reinforcement learning from both AI-generated and human feedback to improve alignment with human expectations and reduce hallucinations.
• Apply instruction tuning and text-to-text task formulation to unify and enhance performance across diverse NLP tasks.
• Train on a diverse, multilingual dataset including Chinese-specific content to improve performance on Chinese-centric benchmarks.
• Utilize a five-shot prompting setting for evaluation to assess zero-shot and few-shot generalization capabilities.

Figure 1: Annotation platform.

Experimental results

Research questions

• RQ1 Can a bidirectional autoregressive architecture reduce memory decay and hallucinations compared to standard autoregressive models?
• RQ2 How does reinforcement learning from both AI and human feedback improve alignment and factual consistency in large language models?
• RQ3 To what extent can parameter expansion from smaller pre-trained models enhance the performance of a large-scale language model like BatGPT?
• RQ4 Does the bidirectional autoregressive design improve zero-shot reasoning and few-shot generalization on complex, multi-domain benchmarks like CMMLU and C-Eval?
• RQ5 How does BatGPT perform on hard, reasoning-intensive subsets of Chinese NLP benchmarks compared to existing models?

Key findings

• BatGPT-15B achieved an average accuracy of 36.72% on the CMMLU benchmark, outperforming all other Chinese-oriented LLMs evaluated.
• On the C-Eval benchmark, BatGPT-15B achieved an overall accuracy of 57.4%, with 72.1% in Social Science and 60.7% in Humanities.
• The model scored 38.48% on the full CMMLU set using direct answer prompting, and 35.91% with chain-of-thought prompting, indicating potential for further improvement with larger scale.
• On the C-Eval Hard subset, BatGPT-15B achieved 36.9% accuracy, demonstrating strong reasoning capabilities on challenging, non-trivial tasks.
• The model showed robust performance across all CMMLU categories, with 37.00% accuracy on China-specific topics and 42.14% on the 'Other' category.
• BatGPT-15B demonstrated consistent performance across different prompting strategies, suggesting strong generalization and adaptability to varied input formats.