[Paper Review] CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models

Jinfeng Zhou, Zhuang Chen+15 more|arXiv (Cornell University)|Nov 28, 2023

Topic Modeling4 citations

TL;DR

CharacterGLM introduces a family of Chinese conversational AI models (6B to 66B parameters) built on ChatGLM, enabling customizable, persona-driven dialogue by configuring character attributes and behaviors. It outperforms major closed-source models like GPT-3.5 in consistency, human-likeness, and engagement, especially in long-turn dialogues, and releases its 6B version and training data for community use.

ABSTRACT

In this paper, we present CharacterGLM, a series of models built upon ChatGLM, with model sizes ranging from 6B to 66B parameters. Our CharacterGLM is designed for generating Character-based Dialogues (CharacterDial), which aims to equip a conversational AI system with character customization for satisfying people's inherent social desires and emotional needs. On top of CharacterGLM, we can customize various AI characters or social agents by configuring their attributes (identities, interests, viewpoints, experiences, achievements, social relationships, etc.) and behaviors (linguistic features, emotional expressions, interaction patterns, etc.). Our model outperforms most mainstream close-source large langauge models, including the GPT series, especially in terms of consistency, human-likeness, and engagement according to manual evaluations. We will release our 6B version of CharacterGLM and a subset of training data to facilitate further research development in the direction of character-based dialogue generation.

Motivation & Objective

• To address the lack of long-term consistency and emotional depth in existing LLMs for social interaction.
• To enable fine-grained customization of AI characters through configurable attributes (identity, interests, viewpoints) and behaviors (linguistic style, emotional expression).
• To develop a system that supports sustained, engaging, and human-like multi-turn dialogues with personalized AI characters.
• To overcome limitations of prompt-tuning and persona-based dialogue in capturing complex, three-dimensional character personalities.
• To release a community-accessible 6B-parameter model and training data to advance research in character-based dialogue systems.

Proposed method

• Fine-tune a series of large language models (6B to 66B parameters) based on ChatGLM to support character-based dialogue (CharacterDial).
• Design a structured profile system to encode character attributes such as identity, interests, experiences, achievements, and social relationships.
• Integrate behavioral control via linguistic features, emotional expression patterns, and interaction styles in response generation.
• Construct a large-scale, diverse dataset of multi-turn dialogues with profile-annotated characters to train the models.
• Apply instruction tuning and preference optimization techniques to align model outputs with desired character behaviors and user engagement.
• Use human evaluation with 10 annotators to compare CharacterGLM against GPT-3.5 and MiniMax across consistency, human-likeness, and engagement.

Figure 1 : Examples of multi-turn character-based dialogue (CharacterDial) between characters customized by CharacterGLM and users. CharacterDial allows users to converse with personalized characters by configuring their profiles.

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM family achieve superior consistency and engagement in long-turn, character-based dialogues compared to mainstream closed-source models?
• RQ2 How effective is profile-based customization in maintaining a coherent, unique character voice across extended dialogues?
• RQ3 To what extent do configurable attributes and behavioral patterns enhance the human-likeness and emotional authenticity of AI-generated responses?
• RQ4 Does the model maintain performance when generating shorter responses, despite preferring longer outputs in human preference studies?
• RQ5 Can a publicly released 6B-parameter version of the model serve as a strong baseline for future research in character-based dialogue?

Key findings

• CharacterGLM-66B outperforms GPT-3.5 in all three evaluation dimensions—consistency, human-likeness, and engagement—achieving a 5% average advantage in overall preference across 10 annotators.
• The model maintains strong performance even when generating shorter responses, despite a general human preference for longer replies, indicating robustness to length variation.
• In pairwise comparisons, CharacterGLM-66B won 49% of overall comparisons in consistency, 48% in human-likeness, and 49% in engagement, with a net advantage of +5% in overall preference.
• The 6B version of CharacterGLM was released alongside a subset of training data to support open research and model fine-tuning in character-based dialogue.
• MiniMax generated longer responses on average, yet CharacterGLM-66B still achieved comparable or better preference scores, indicating high quality per token.
• The study identifies long-term memory, self-awareness, character society interaction, and cognitive process modeling as key challenges for future development of AI characters.

(a) Pair-wise comparison results of CharacterGLM over GPT-3.5 in consistency, human-likeness, and engagement.